AWARD  NUMBER: 


W81 XWH-1 5-1  -0403 


TITLE:  Development  of  an  Integrated  Team  Training 

Design  and  Assessment  Architecture  to  Support 
Adaptability  in  Healthcare  Teams 


PRINCIPAL  INVESTIGATOR:  Rosemarie  Fernandez,  MD 


CONTRACTING  ORGANIZATION:  University  of  Washington 

Seattle,  WA  98195 


REPORT  DATE: 
TYPE  OF  REPORT: 


October  2017 
Annual 


PREPARED  FOR:  U.S.  Army  Medical  Research  and  Materiel  Command 

Fort  Detrick,  Maryland  21702-5012 


DISTRIBUTION  STATEMENT:  Approved  for  Public  Release;  Distribution  Unlimited 


The  views,  opinions  and/or  findings  contained  in  this  report  are  those  of  the  author(s)  and  should  not 
be  construed  as  an  official  Department  of  the  Army  position,  policy  or  decision  unless  so  designated 
by  other  documentation. 


REPORT  DOCUMENTATION  PAGE 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and  maintaining 
the  data  needed,  and  completing  and  reviewing  this  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information,  including  suggestions  for 
reducing  this  burden  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports  (0704-0188),  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington, 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  any  penalty  for  failing  to  comply  with  a  collection  of  information  if  it  does  not 
display  a  currently  valid  OMB  control  number.  PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 

1.  REPORT  DATE  2.  REPORT  TYPE 

October  2017  Annual 

3.  DATES  COVERED 

30  Sep  2016-  29  Sep  2017 

4.  TITLE  AND  SUBTITLE 

Development  of  an  Integrated  Team  Training  Design  and  Assessment 
Architecture  to  Support  Adaptability  in  Healthcare  Teams 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

W81 XWH-1 5-1  -0403 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

Rosemarie  Fernandez,  MD 

James  Grand,  PhD 

E-Mail:  fernanre(a)uw.edu  ;  qrandiam(a)umd.edu 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

UNIVERSITY  OF  WASHINGTON 

4333  BROOKLYN  AVE  NE 

SEATTLE  WA  98195-0001 

8.  PERFORMING  ORGANIZATION 

REPORT 

NUMBER 

9.  SPONSORING  /  MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

U.S.  Army  Medical  Research  and  Materiel  Command 

Fort  Detrick,  Maryland  21702-5012 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION  /  AVAILABILITY  STATEMENT 

Approved  for  Public  Release;  Distribution  Unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

Purpose  and  Scope:  It  is  the  purpose  of  this  project  to  optimize  adaptability  and  mitigate  teamwork-related  threats  to 
patient  safety  by  addressing  key  methodological  and  conceptual  gaps  in  healthcare  simulation-based  team  training.  The 
investigators  are  developing  the  necessary  conceptual  framework  and  team  performance  assessment  mechanisms  to 
support  training  systems  that  improve  adaptability  and  performance  in  trauma  teams. 

Aim  la.  Develop  a  team  training  design  architecture  to  support  simulation-based  training/assessment  systems  capable  of 
developing  adaptive  expertise  in  healthcare  teams 

Aim  1  b.  Develop  evidence-based  guidelines  and  recommendations  for  the  development  of  embedded,  adaptive  feedback 
and  performance  assessments 

Aim  2.  Develop  and  refine  a  predictive  model  of  trauma  team  performance  and  outcomes  for  use  in  an  adaptive 
guidance  system 

Major  Findings:  The  investigators  performed  a  four-step  process  to  develop  a  unified  team  training  design  architecture  and 
supporting  conceptual  framework.  They  identified  key  training  design  principles  and  recommendations  for  the  development 
and  implementation  of  embedded,  adaptive  feedback  and  performance  assessment.  The  investigators  designed  a 
prototype  of  a  Bayesian  Belief  Network  (BBN)-based  model  of  trauma  team  performance  and  outcomes.  The  investigators 
went  through  several  model  iterations  and  settled  on  one  model  for  final  testing.  This  testing  and  validation  (Aim  2)  is  the 
focus  of  the  NCE  year. 

Impact:  The  provision  of  emergency  care  in  a  combat  situation  mandates  well-developed  adaptive  expertise,  making  this 
work  relevant  to  military  healthcare.  Our  work  provides  a  roadmap  and  mechanism  for  future  work  in  a  multitude  of 
healthcare  teams  and  settings. 

15.  SUBJECT  TERMS 

Military  healthcare  team;  Trauma  teams;  Team  training;  Teamwork;  Adaptive  performance;  Leadership; 
Simulation;  Modeling;  Bayesian  belief  networks  (BBN) 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION 

OF  ABSTRACT 

18.  NUMBER 
OF  PAGES 

19a.  NAME  OF  RESPONSIBLE 
PERSON 

USAMRMC 

a.  REPORT 

Unclassified 

b.  ABSTRACT 

Unclassified 

c.  THIS  PAGE 

Unclassified 

Unclassified 

50 

19b.  TELEPHONE  NUMBER  (include 
area  code) 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std.  Z39.18 


Table  of  Contents 


Page 


1.  Introduction .  5 

2.  Keywords .  5 

3.  Accomplishments .  6 

4.  Impact .  21 

5.  Changes/Problems .  22 

6.  Products .  23 

7.  Participants  &  Other  Collaborating  Organizations .  24 

8.  Special  Reporting  Requirements .  25 

9.  Attachments .  26 


1.  INTRODUCTION 

Health  care  team  performance  is  critical  to  the  provision  of  safe,  efficient,  and  effective  care.  Team  adaptability 
is  necessary  for  effective  team  performance  and  is  especially  critical  for  trauma  teams,  whose  members  must 
anticipate  change  and  rapidly  coordinate  effective  responses.  Teams  that  are  not  highly  adaptive  function  in  a 
reactive  mode  that  is  fraught  with  potential  safety  and  error  risks.  Rigorously  designed  computer-based 
simulation  systems  have  the  potential  to  support  active  learning  experiences  and  improve  adaptability  and 
performance  in  individuals  and  teams.  This  technology  has  the  potential  to  link  individuals,  teams,  and  units 
together  for  the  purpose  of  engaging  in  common  training  exercises.  However,  without  the  proper  supporting 
design  elements,  these  simulations  are  ineffective  and  inefficient  training  tools.  Current  health  care  team 
training  models  and  strategies  do  not  specifically  leverage  the  training  design  elements  and  assessment-driven 
feedback  mechanisms  that  improve  team  performance  in  highly  dynamic  settings.  The  goal  of  the  proposed 
project  is  to  improve  health  care  team  adaptability  and  patient  safety  by  providing  the  necessary 
conceptual  framework  and  assessment  mechanism  to  support  the  design  and  implementation  of  highly 
effective  simulation-based  team  training  with  embedded,  adaptive  guidance.  This  project  is  organized  into  the 
following  Aims: 

Aim  la.  Develop  a  team  training  design  architecture  to  support  simulation-based  training 

/assessment  systems  capable  of  developing  adaptive  expertise  in  healthcare  teams 

Aim  1b.  Develop  evidence-based  guidelines  and  recommendations  for  the  development  of 
embedded,  adaptive  feedback  and  performance  assessments 

Aim  2.  Develop  and  refine  a  predictive  model  of  trauma  team  performance  and  outcomes  for  use  in 
an  adaptive  guidance/feedback  system 

The  outcomes  from  this  research  will  provide  the  fundamental  knowledge,  both  conceptual  and  operational,  to 
support  the  development  of  simulation-based  team  training  systems  with  embedded  guidance.  Our  long-term 
goal  is  to  optimize  health  care  team  performance  and  adaptability  through  rigorous  training  design. 


2.  KEYWORDS  AND  ABBREVIATIONS 

Healthcare  team 
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Trauma  teams 

Team  training 
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Adaptability 

Adaptive  performance 
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Simulation 

Modeling 
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3.  ACCOMPLISHMENTS 


3a.  What  were  the  major  goals  of  the  project  (organized  by  Aim)? 

Aim  la.  Develop  a  team  training  design  architecture  to  support  simulation-based  training/assessment 
systems  capable  of  developing  adaptive  expertise  in  health  care  teams 
Aim  1b.  Develop  evidence-based  guidelines  and  recommendations  for  the  development  of  embedded, 
adaptive  feedback  and  performance  assessments 

The  primary  outcome  of  Aim  la  is  a  conceptually  and  methodologically  sound  training  design  architecture  that 
supports  the  development  and  integration  of  team  training  and  automated  assessment  technologies  in 
simulation  environments.  The  primary  outcome  of  Aim  1b  is  a  set  of  best  practice  guidelines  and 
recommendations  for  the  design  and  incorporation  of  adaptive,  embedded  feedback  (guidance)  into  simulation- 
based  team  training.  The  tasks,  timeline,  and  status  of  each  step  associated  with  Aims  la  and  1b  are 
summarized  in  the  table  below. 

This  Aim  is  100%  completed,  with  manuscript  preparation  and  dissemination  underway.  We  will  revisit  and 
update  concepts  and  principles  based  upon  information  gained  in  Aim  2.  The  work  proposed  is  complete. 


Aims  la  and  1b:  Major  Goals  and  Tasks 


Aims  la  and  1b  Tasks 

Timeline 

(Months) 

Status 

Task  1 :  Project  Start-up 

Establish  subcontracts  to  enable  purchasing. 

0-3 

Completed 

Local/Site  IRB  application  submissions 

0-3 

All  IRB  submissions  have  been  completed  and  the  project  has 
been  awarded  exempt  status  by  each  institution.  Completed 

Assembly  of  subject  matter  expert  panel 

0-3 

Subject  matter  experts  have  been  invited  and  the  panel  now 
contains  experts  from  emergency  medicine,  simulation,  trauma 
surgery,  and  nursing.  Individuals  were  chosen  for  their 
expertise  and  to  ensure  geographical  representation. 

Completed 

Human  Research  Protection  Office  IRB 

3 

The  HRPO  has  granted  exempt  status.  Completed 

Milestone(s)  Achieved: 

1 .  Project  infrastructure  in  place 

2.  Local/Site  IRB  and  HRPO  Approval 

6 

100%  COMPLETED 

Task  2:  Identify  constructs  of  interest 

Literature  search  strategy 

0-3 

Search  strategy  within  healthcare  literature,  trauma 
performance  literature,  trauma  outcomes  literature,  and  team 
science  has  been  defined.  Completed 

Review  of  identified  manuscripts  and 
literature 

0-6 

The  review  of  relevant  literature  (healthcare  and  team  science) 
to  inform  the  conceptual  model  and  framework  of  adaptive 
performance  has  been  completed.  Completed 

Milestone(s)  Achieved: 

1 .  Identification  of  individual  and  team 
performance  constructs  for  the 
conceptual  framework  and  training 
architecture 

6 

We  identified  relevant  individual  and  team  constructs  and 
designed  a  draft  framework.  We  anticipate  continuing  to  revisit 
this  framework  as  model  testing  occurs  (Aim  2).  We  show  this 
as  an  ongoing  milestone  nearly  complete. 

ON  TIME,  99%  COMPLETED 

Task  3:  Determine  relevant  variables  and  relationships 

Develop  nomological  net  among  constructs 
identified  in  Task  2 

3-9 

We  have  identified  key  relationships  between  processes  and 
variables  critical  for  team  adaptability.  Completed 

Subject  matter  expert  review  of  variables 
and  relationships 

6-9 

Trauma  care  and  military  experts  reviewed  the  components  of 
our  adaptability  model.  Modifications  included  the  addition  of 
cognitive  adaptability  and  diagnostic  process  as  a  key 
component  of  trauma  team  adaptive  capacity.  Completed 

Milestone(s)  Achieved: 

1 .  Identification  of  relationships  between 
individual  and  team  performance 
constructs  for  the  conceptual 
framework  and  training  architecture 

9 

99%  COMPLETED 

We  indicate  this  milestone  as  near  completed  as  we  intend  to 
use  relevant  data  from  Aim  2  to  refine  our  work 

Task  4:  Identify  appropriate  level  of  constructs  and  variables 

Identification  of  appropriate  levels  for 
constructs,  relationships,  and  outcomes 
identified  in  Task  3 

6-9 

Literature  reviews  and  subject  matter  expert  opinion  was  used 
to  choose  and  adapt  a  model  of  individual,  team,  and  system- 
level  measurement  necessary  to  guide  the  development  and 
implementation  of  effective  team  training.  Completed 

Milestone(s)  Achieved: 

1 .  Multilevel  framework  of  healthcare 
team  training  performance 

9 

We  identified  relevant  individual  and  team  constructs  and 
designed  a  draft  framework.  We  anticipate  continuing  to  revisit 
this  framework  as  model  testing  occurs;  therefore  will  reflect 
this  as  an  ongoing  milestone  nearly  complete. 

ON  TIME,  99%  COMPLETED 

Task  5:  Identify  appropriate  outcome  measures  and  mechanisms 

Construct  framework  for  provision  of 
adaptive  guidance  during  simulation-based 
team  training 

6-9 

Relevant  feedback  mechanisms  and  designs  have  been 
identified  and  a  draft  framework  has  been  designed.  We 
anticipate  revising  the  feedback  mechanisms  and  design 
based  upon  Aim  2.  We  therefore  reflect  this  task  as  On  time, 
99%  completed. 

Subject  matter  expert  review  of  feedback 
framework 

9-12 

Our  military,  external  team  science,  and  external  emergency 
medicine  subject  matter  experts  reviewed  the  structure  of  our 
feedback  framework  to  ensure  the  framework  is  compatible 
with  current  military  training  efforts  and  reflective  of  current 
team  science  recommendations. 

Completed 

Milestone(s)  Achieved: 

1 .  Integrated  team  training  design 
architecture 

2.  Evidence-based  guidelines  and 
recommendations  for  the  provision  of 
embedded,  adaptive  guidance 

12 

As  noted  we  will  revisit  recommendations  and  principles  based 
upon  findings  in  Aim  2.  We  therefore  reflect  this  milestone  as 

99%  COMPLETED 

See  Attachments  1  -  3,  remainder  of  material  was  provided  in 
the  2016  annual  report. 

Task  5a:  Cross  reference  feedback  principles  and  team  training  architecture  with  TeamSTEPPS  terminology 
(ADDITIONAL  TASK  ADDED  TO  ADDRESS  IPR) 

Review  current  terminology  and  link  both 
feedback  principles  and  training 
architecture  with  TeamSTEPPS  principles 
and  trainer  materials 

18 

This  work  was  not  initially  proposed  but  was  added  in  response 
to  the  IPR  comments.  We  completed  this  work  and  provide 
these  materials  in  Attachments  1  -  3.  Completed 

Review  current  terminology  and  ensure 
Crawl-Walk-Run  terminology  is 
incorporated  and  clearly  highlighted  for 
instructors. 

18 

This  work  was  not  initially  proposed  but  was  added  in  response 
to  the  IPR  comments.  We  completed  this  work  and  provide 
these  materials  in  Attachments  1  -  3.  Completed 

Milestone(s)  Achieved: 

1 .  Developed  a  glossary  of  terms  linking 
feedback  guidelines  and  training 
architecture  with  TeamSTEPPS 
components. 

18 

Please  see  Attachments  1-3,  we  now  note  where 

TeamSTEPPS  concepts  fit  within  our  framework. 

100%  COMPLETED 

Aim  2.  Develop  and  refine  a  predictive  model  of  trauma  team  performance  and  outcomes  for  use  in 
an  adaptive  guidance/feedback  system 

The  primary  outcome  from  Aim  2  is  a  predictive  trauma  team  performance  assessment  tool  that  generalizes  to 
teams  of  varying  expertise  levels  and  across  civilian  and  military  contexts  and  is  capable  of  supporting 
embedded,  adaptive  guidance  during  simulation-based  team  training.  Our  approach  examines  the  use  of 
Bayesian  Belief  Networks  (BBNs)  to  support  the  provision  of  adaptive,  embedded  guidance  that  facilitates 
development  of  adaptive  expertise  and  trauma  team  performance.  We  utilize  existing  simulation-based  trauma 
team  performance  data  to  construct  a  BBN  that  models  the  relationships  between  key  individual  and  team 
characteristics,  behavioral  outcomes,  and  patient  care  events  in  a  previously  well-defined  and  validated 
simulated  scenario.  The  model  will  leverage  the  probabilistic  interdependencies  among  these  variables  to 
enable  educators  and/or  learners  to  assess  the  likelihood  of  critical  team/patient  outcomes  in  the  simulated 
environment.  We  then  incorporate  the  design  architecture  conceptual  foundations  developed  in  Aims  la&b  to 
guide  the  transformation  of  predictive  model  data  into  an  adaptive  guidance  tool.  The  tasks,  timeline,  and 
status  of  each  step  associated  with  Aim  2  are  summarized  in  the  table  below. 


Aim  2:  Major  Goals  and  Tasks 


SPECIFIC  AIM  2 

Timeline 

(Months) 

Status 

Task  6:  Collection  of  prospective  simulation  data 

Subject  recruitment 

4-6 

Completed,  100%  completed 

Execute  trauma  resuscitation  simulations 

4-6 

We  have  completed  the  simulations  necessary  for  the  study; 
however,  we  wish  to  maximize  the  inclusion  of  military 
personnel  and  therefore  will  continue  to  enroll  military  providers 
through  September  2017.  This  will  extend  enrollment  and  is 
one  reason  we  are  requesting  a  NCE.  No  additional  funds  will 
be  required  to  complete  this  work. 

95%  completed 

Train  and  calibrate  raters 

6 

Rater  training  has  been  designed  to  code  new  simulations. 
Existing  trauma  videos  have  been  coded,  with  excellent  inter¬ 
rater  reliability.  We  anticipate  refresher  training  periodically. 

Ongoing,  Initial  work  100%  completed. 

Code  videos  of  simulated  resuscitations 
using  patient  care  and  teamwork  measures 

6-12 

Simulation  video  processing  has  slightly  delayed  the  initiation  of 
coding;  coding  is  now  underway.  To  ensure  timely  completion, 
we  have  hired  additional  video  processors  and  purchased 
additional  storage  to  allow  more  rapid,  efficient  video 
processing.  With  continued  enrollment  of  military  providers,  this 
work  will  continue  until  all  simulations  are  completed.  This  will 
extend  into  the  NCE  period. 

Delayed,  90%  complete 

Transform  data  into  appropriate  categorical 
structure  for  BBN 

9-12 

We  completed  initial  transformation  of  existing  data  into  a 
categorical  structure.  This  is  required  to  execute  BBN  modeling 
and  requires  the  input  of  clinical  experts.  Based  on  this  data 
transformation,  an  initial  structure  for  the  BBN  was  constructed 
using  the  transformed  data  (see  also  Task  8).  Choices  about 
data  discretization  and  model  structure  offer  different 
advantages  that  the  research  team  continually  evaluates,  so  we 
anticipate  this  process  being  an  iterative  one  as  we  proceed 
through  Tasks  7  and  8. 

A  second  version  of  the  BBN  has  been  developed  based  on 
behavioral  clusters  identified  from  existing  simulation-based 
resuscitation  performance  data.  This  prototype  BBN  has  been 
entered  into  the  Netica  software  for  further  analysis. 

Overall,  this  subtask  is  part  of  an  iterative  process  with 
refinement  of  the  model  occurring  throughout  Aim  2.  We 
therefore  reflect  this  step  as  99%  Completed 

Milestone(s)  Achieved: 

1.  Team  data  set  of  teamwork  and 
patient  care  performance  during 
trauma  resuscitation  simulation 

12 

In  Progress,  90%  COMPLETE 

(This  work  is  ongoing,  as  refining  the  model  based  on  new 
results  is  part  of  the  development  process) 

Task  7:  Identify  and  define  variables  (nodes)  for  inclusion  in  team  assessment  model 

Examination  of  conceptual  frameworks 
and  literature  review  from  Aims  la  and  1b 

9-12 

We  have  finalized  the  review  of  feedback  principles  to  make 
final  decisions  regarding  when  the  BBN  will  be  designed  to 
provide  information  to  learners  and  instructors  and  in  what 
format  the  feedback  should  be  delivered.  This  subtask  was 
delayed  by  approximately  1  month  and  is  now  completed. 
Completed 

Evaluation  of  existing  experimental 
dataset  to  identify  and  extract  variables  of 
interest 

9-12 

We  have  completed  review  of  all  existing  datasets.  This  process 
resulted  in  identification  of  -150  usable  variables  for  which  data 
is  available.  We  developed  protocols  for  evaluating 
inclusion/selection  of  items  as  variables  in  the  BBN.  An  initial 
protocol  was  used  to  guide  the  development  of  the  first 
prototype  BBN  (see  also  Tasks  6,  8),  and  an  additional  protocol 
has  been  developed  to  guide  the  development  of  a  second 
version  of  the  BBN.  This  task  is  on  time  and  completed  for  the 
initial  BBN  and  revised  BBN.  Prior  to  final  dissemination  this 
step  will  be  revisited;  however,  the  proposed  work  is  completed. 
We  therefore  reflect  this  as  99%  Completed 

Milestone(s)  Achieved: 

1 .  Identification  of  observable  measures 
and  latent  constructs  to  be 
incorporated  into  the  BBN 

12 

99%  COMPLETED 

(This  milestone  is  largely  completed,  delays  reflect  extended 
subject  enrollment  and  the  iterative  process  of  BBN 
development) 

Task  8:  Design  the  structure  for  the  prototype  BBN  team  assessment  system 

Identify  appropriate  and  parsimonious 
candidates  for  the  causal  structure  among 
the  variables 

12-15 

We  have  developed  and  evaluated  multiple  possible 
organizational  structures  for  BBNs.  We  provided  an  initial  draft 
of  these  in  the  annual  report  submitted  11/2016  and  an  updated 
draft  on  06/30/2017.  The  research  team  since  identified  and 
evaluated  three  alternative  structures  of  the  BBN  based  on  utility 
of  application.  The  most  recent  BBN  structure  (described  further 
below)  has  been  applied  to  model  all  events  during  a  trauma 
simulation.  Relationships  represented  in  the  BBN  will  likely  be 
iterated  as  the  remainder  of  the  simulation  is  modeled,  but  the 
foundational  work  is  completed. 

99%  completed,  (work  is  iterative) 

Subject  matter  expert  review  of  variable 
relationships 

12-15 

To  facilitate  a  thorough  and  comprehensive  SME  review, 
including  military  experts,  the  research  team  decided  to  hold  the 
review  after  the  final  prototype  of  the  model  is  completed.  As  a 
result,  this  subtask  will  be  accomplished  during  Q9  as  part  of 
finalizing  the  prototype  prior  to  testing. 

0%  completed,  (Q9  completion  planned) 

Milestone(s)  Achieved: 

1 .  Identification  of  multiple  candidate 

BBNs  for  the  observed  variables 

15 

DELAYED,  80%  COMPLETED  (Q9  completion  planned) 

Task  9:  Generate  initial  probability  tables  for  BBN  team  assessment  system 

Transform  data  into  appropriate 
categorical  structure 

12-15 

Based  on  the  proposed  BBN  structure  developed  in  Task  8,  we 
transformed  the  dataset  into  categorical  structures  that  logically 
developed  based  upon  the  clinical  content  and  the  BBN 
prototype  structure.  Choices  about  how  to  discretize  certain  data 
and/or  whether  existing  simulations  should  be  recoded  to 
facilitate  the  model  completion  will  be  evaluated  as  the  final 

BBN  prototype  is  completed.  This  work  is  completed,  but  will  be 
revisted  after  testing  the  BBN  with  the  existing  dataset. 

99%  COMPLETED 

Explore  different  learning  algorithms 

15-18 

This  work  will  begin  in  earnest  using  the  revised  version  of  the 
BBN.  The  initial  version  was  a  prototype  and  did  not  contain  the 
structural  data  elements  necessary  for  testing.  We  anticipate 
that  this  work  will  be  completed  in  the  first  quarter  of  the  NCE 
year. 

Delayed,  0%  completed 

Assess  BBN  fit 

15-18 

This  work  will  begin  in  earnest  using  the  revised  version  of  the 
BBN.  The  initial  version  was  a  prototype  and  did  not  contain  the 
structural  data  elements  necessary  for  testing.  We  anticipate 
that  this  work  will  be  completed  in  the  first  quarter  of  the  NCE 
year. 

Delayed,  0%  completed 

Generate  conditional  dependencies  for 
unavailable  data 

15-18 

This  work  will  begin  in  earnest  using  the  revised  version  of  the 
BBN.  The  initial  version  was  a  prototype  and  did  not  contain  the 
structural  data  elements  necessary  for  testing.  We  anticipate 
that  this  work  will  be  completed  in  the  first  quarter  of  the  NCE 
year. 

Delayed,  0%  completed 

(NEW  SUBTASK)  Recode  existing  videos 
to  provide  additional  data  to  provide 
additional  nodes  and  data  for  the  BBN 

24-30 

This  subtask  was  added  based  upon  early  drafts  of  the  BBN. 

We  identified  additional  items  that  would  support  the  BBN 
development  and  make  our  process  more  adaptable  to  other 
trauma  care  events. 

New  Subtask,  0%  completed 

Milestone(s)  Achieved: 

1 .  Functional  prototype  BBN  team 
assessment  system 

18 

Ongoing,  30%  completed 

Task  10:  BBN  team  assessment  system  calibration 

Transform  prospective  data  into 
appropriate  categorical  structure  for  BBN 

12-15 

This  process  will  be  performed  as  part  of  the  data  analysis  from 
prospective  data  collection.  As  data  coding  is  not  complete,  this 
work  has  been  delayed.  Flowever,  the  transformations  that 
have  been  performed  on  existing  data  will  provide  a  clear 
roadmap  for  this  process  and  we  anticipate  no  problems  with 
this  work  based  upon  current  data  transformations.  This  work 
will  occur  in  the  second  quarter  of  the  NCE  year. 

Delayed,  0%  complete 

Use  prospectively  collected  data  to 
calibrate  BBN 

18-21 

This  work  is  also  delayed  and  will  follow  the  above  subtask,  will 
be  completed  in  the  second  quarter  of  the  NCE  (PY3)  year. 

Delayed,  0%  completed 

Use  subject  matter  experts  and  empirical 
data  from  the  literature  review  in  Aim  la  to 
adapt  the  BBN  as  needed 

18-21 

This  work  will  naturally  follow  the  prospective  data  collection  and 
will  form  the  final  step  of  the  project.  We  anticipate  this  work  will 
occur  in  the  third  quarter  of  the  NCE  year  (PY3). 

Delayed,  0%  completed 

Milestone(s)  Achieved: 

1 .  Functional,  generalizable  prototype 

BBN  trauma  team  assessment  system 

21 

Delayed,  0%  completed 

Task  11:  Report  writing  and  dissemination 

Prepare  final  report  and  manuscripts 

21  -24 

Planned,  0%  completed 

Submit  final  reports  and  manuscripts 

24 

Planned,  0%  completed 

Milestone(s)  Achieved: 

1 .  Dissemination  of  methodological 
approach  and  empiric  findings 

24 

Planned,  0%  completed 

BBN  =  Bayesian  Belief  Network 


3b.  What  was  accomplished  under  these  goals  (organized  by  Aim)? 


Aim  la.  Develop  a  team  training  design  architecture  to  support  simulation-based  training/assessment 
systems  capable  of  developing  adaptive  expertise  in  health  care  teams 
Aim  1b.  Develop  evidence-based  guidelines  and  recommendations  for  the  development  of  embedded, 
adaptive  feedback  and  performance  assessments 


Data  Collection:  A  robust  literature  review  is  critical  to  the  development  of  a  comprehensive  health  care  team 
training  design  architecture.  We  conducted  an  extensive  literature  review,  both  within  healthcare  and  team 
science  literature  to  identify  key  components  of  team  performance  adaptability.  We  focused  specifically  on 
identifying  the  individual  and  team  processes  that  drive  adaptive  behaviors,  as  well  as  possible  metrics  that 
would  indicate  adaptability  at  individual  and  team  levels.  We  then  convened  a  multidisciplinary  group  of  nurses 
and  physicians  from  both  civilian  and  military  health  care  settings  to  provide  expertise  and  insight  into  how 
these  adaptive  behaviors  translate  to  the  health  care  setting,  and  how  they  might  develop  over  different  levels 
of  expertise.  Finally,  we  observed  both  simulated  and  actual  trauma  team  performance  to  augment  our  data 
and  further  our  understanding  of  how  adaptive  performance  unfolds  during  highly  complex  clinical  activities. 
This  information  was  then  used  to  inform  the  identification  of  key  conceptual  models  described  below. 

Defining  Adaptive  Performance  in  Trauma  Teams:  We  used  the  literature  review  and  subject  matter  expert 
review  described  above  to  identify  all  individual  and  team-performance  concepts  and  constructs  that  are 
relevant  to  training,  assessing,  and  supporting  adaptive  trauma  team  performance.  Our  initial  adaptive 
performance  model  did  not  reflect  the  need  for  trauma  teams  to  rapidly  incorporate  new  diagnostic  information 
into  the  team’s  plans  and  processes.  Subject  matter  experts  raised  an  issue  that  cognitive  processes  were  not 
adequately  represented.  We  therefore  reviewed  the  diagnostic  error  literature,  diagnostic  decision-making 
literature,  and  team  learning  research  to  augment  our  model.  The  result  is  listed  in  Figure  1. 


Figure  1 . 
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This  model  reflects  the  cognitive  and  behavioral  process  components  of  trauma  team  performance.  First, 
cognition  is  represented  by  the  team’s  efforts  to  make  sense  of  the  situation  (Situation  Assessment).  Briefly, 
the  team  must  use  existing  data/observations  to  identify  the  patient-  and  team-related  tasks  and  demands. 
This  information  is  then  used  to  develop  a  differential  diagnosis.  Based  on  this/these  diagnoses,  the  team  has 


expectations  regarding  how  the  patient  will  respond  to  treatments  and  how  his/her  condition  will  evolve  over 
time.  The  team  continuously  compares  this  “expected”  state  to  the  “observed”  state  of  the  patient.  This 
comparison  informs  the  team  and  helps  regulate  the  team  processes  that  regulate  task  performance.  If  the 
team  notes  a  mismatch  between  expected  patient  improvement  and  current  patient  condition,  this  should 
prompt  the  team  to  review  their  plan,  make  adjustments,  and  execute  the  modified  plan.  The  results  of  these 
new  actions  should  be  monitored  and  evaluated.  The  observations  made  during  evaluation  become  the 
information  that  the  team  uses  to  reassess  the  situation,  reconsider  the  differential  diagnosis(es),  and  the 
adaptive  cycle  continues.  In  a  rapidly  evolving  trauma  resuscitation,  this  cycle  repeats  continuously  to  ensure 
the  team  is  adapting  to  the  unstable  patient/team/environment. 

Identifying  appropriate  training  targets:  Training  should  be  purposeful  and  should  target  appropriate 
cognitive,  behavioral,  and  affective/motivational  processes  in  a  stepwise  fashion.  Training  mechanisms  should 
support  both  skill  implementation  in  the  clinical  environment  as  well  as  transfer  to  novel  situations.  We 
identified  a  staged  approach  to  training  that  targets  appropriate  skills  necessary  to  develop  adaptive  capacity. 
We  include  both  individual  and  team-based  processes  as  well  as  training  mechanisms.  The  framework  below 
(Figure  2)  provides  an  outline  for  this  approach. 


Figure  2.  Training  targets  and  training  techniques 
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Identifying  appropriate  level  of  constructs  and  variables:  A  thorough  understanding  of  individual  and  team 
performance  within  complex  environments  necessitates  a  multilevel  approach  to  theory-building  and  outcomes 
research.  Organization-level  phenomena  emerge  through  the  behavior,  perceptions,  affect,  and  interactions  of 
individuals  and  team.  Likewise,  individuals  and  teams  are  directly  influenced  by  the  culture,  norms,  and 
structure  of  the  organization.  Ignoring  the  multilevel  nature  of  a  construct,  intervention,  or  relationship  may 
result  in  oversimplification  of  outcomes  and  failure  to  recognize  important  measurement  targets.  We 
developed  a  multilevel  conceptual  architecture  of  adaptation  that  considers  (1)  the  types  of  events  teams  must 
adapt  to  (i.e.,  what  type  of  change  is  occurring),  (2)  the  types  of  processes  teams  use  to  adapt,  and  (3)  at  what 
level  these  processes  occur.  This  taxonomy  (Figure  3)  can  help  guide  the  selection  of  appropriate  training 
targets  and  can  help  educators  target  correct  task  complexity,  appropriate  processes 
(cognitive/behavioral/affective),  and  direct  training  and  measurement  at  the  correct  level  (individual,  team, 
unit).  Such  specificity  is  important,  as  being  purposeful  when  designing  training  will  ensure  that  individuals, 
teams,  and  units  are  prepared  for  the  specific  types  of  adaptation  necessary  for  their  work.  This  level  of 
specificity  in  training  is  often  overlooked  and  is  not  part  of  current  training  guidelines.  In  Attachment  1  we 
describe  training  principles  related  to  (1)  level  of  training  and  (2)  specific  processes  targeted  by  training.  In 
Attachment  2  we  then  describe  three  different  task  requirements  for  adaptability  and  specifically  identify 
training  principles  associated  with  each  type  of  task  complexity. 


Figure  3.  Model  of  task  complexity,  processes,  and  level(s)  of  analysis 
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Identifying  appropriate  outcome  measures  and  mechanisms:  We  noted  that  training  evaluation  systems 
should  consider  both  proximal  and  distal  outcomes.  Proximal  outcomes  include  both  learning  and 
performance-based  outcomes  and  can  include  basic  declarative  knowledge  as  well  as  more  complex  strategic 
knowledge  and  performance.  Distal  outcomes  that  are  trainee-focused  include  the  transfer  of  learned  skills  to 
the  work  (clinical)  environment  as  well  as  the  application  of  learned  skills  to  novel  situations,  i.e.,  adaptability. 
High-level  distal  outcomes  include  patient,  system,  and  organization-level  outcomes.  Our  literature  review 
focused  on  the  identification  of  pertinent  proximal  and  distal  outcomes.  We  considered  our  own  systematic 
reviews  as  well  as  other  health  care  team  reviews  to  determine  the  current  state  of  team  assessment.  We 
extended  this  knowledge  by  investigating  the  team  science,  safety  science,  and  human  factors  literature. 
Because  our  work  focuses  on  developing  adaptive  expertise,  considerable  efforts  were  made  to  identify 
outcome  measures  that  reflect  adaptive  capacity.  Subject  matter  expert  review  was  utilized  to  help  identify 
where  non-health  care  team  assessments  can  be  translated  into  appropriate  health  care  team  training 
evaluation  targets.  In  Figure  4  we  propose  a  translational  simulation-based  research  model  that  considers 
appropriate  outcome  measures  and  relationships  for  individual  and  team-level  adaptability. 


Figure  4.  Multilevel  outcome  model  for  training  evaluation 
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Recommendations  for  the  provision  of  adaptive  feedback:  For  the  purposes  of  this  work,  we  considered 
(1)  performance  measures  used  for  the  provision  of  feedback  and  (2)  training  evaluation/outcome  metrics  used 


to  measure  training  impact,  separately.  The  provision  of  feedback  is  a  major  focus  of  this  study,  with  the  goal 
of  developing  an  assessment  system  capable  of  supporting  embedded,  adaptive  guidance.  We  therefore 
directed  our  efforts  towards  developing  a  conceptual  framework  to  support  the  content,  structure,  and  provision 
of  adaptive  guidance  during  trauma  team  simulations.  This  work  relied  heavily  on  the  training,  education,  and 
debriefing  literatures.  In  Attachment  3  we  list  feedback  principles,  scientific  rationale,  and,  where  appropriate, 
exemplars  for  simulation-based  training. 

Cross  reference  feedback  principles  and  team  training  architecture  with  TeamSTEPPS  terminology: 

The  investigators  attended  the  2016  IPR  held  in  Fort  Detrick,  MD.  There,  they  presented  preliminary  work  and 
received  constructive  feedback  both  in  person  and  via  written  review.  Since  the  IPR,  the  investigators 
addressed  each  point  made  by  the  panel  and  specific  comments  made  by  COL.  Hopkins-Chadwick  during  a 
phone  meeting.  We  added  an  additional  item  to  our  task  list  (Task  5a)  that  we  feel  clarifies  our  work  and 
improve  usability  by  military  units.  This  task  has  since  been  completed  and  Attachments  1-3  reflect  these 
modifications.  This  completes  our  response  to  the  Panel  and  COL.  Hopkins-Chadwick. 

Deliverables:  We  are  preparing  a  manuscript  for  submission  in  the  healthcare  literature.  Planned  targets 
include  Academic  Medicine  and  Academic  Emergency  Medicine. 

Aim  2.  Develop  and  refine  a  predictive  model  of  trauma  team  performance  and  outcomes  for  use  in  an 
adaptive  guidance/feedback  system 

Trauma  Simulations  and  Performance  Coding:  The  purpose  of  conducting  trauma  team  simulations  is  to 
provide  baseline  data  for  the  design  of  the  BBN.  These  simulations  will  be  used,  along  with  existing  simulation 
data,  to  inform  the  structure  of  the  BBN.  Subject  recruitment  for  prospective  trauma  simulations  is  completed, 
and  the  simulation  sessions  are  underway.  We  have  identified  and  recruited  both  civilian  and  military  trauma 
team  leaders.  To  date  we  have  conducted  all  of  the  proposed  simulations  (n=20).  We  will  continue  to  perform 
simulations  involving  military  providers  to  ensure  adequate  representation  but  have  collected  the  data 
necessary  to  move  ahead  with  the  BBN  design  and  testing. 

BBN  Structure:  We  explored  several  candidate  approaches  to  BBN  design.  The  overall  structure  determined 
to  be  most  informative  for  the  purposes  of  the  project  is  summarized  in  Figure  5.  Briefly,  the  adaptive 
performance  model  presented  in  Figure  1  was  used  to  identify  three  core  activities  relevant  to  team  adaptation: 
(1)  information  gathering  (encompassing  situation  assessment  activities  relevant  to  formulating/revising 
diagnoses  and  establishing  goals  and  team  regulation  activities  related  to  monitoring  and  evaluating  team 
actions/progress);  (2)  communication  (encompassing  team  regulation  activities  relevant  to  planning,  preparing, 
and  coordinating  team  behavior);  and  (3)  action  (encompassing  team  regulation  activities  relevant  to  making 
decisions  and  carrying  out  task  activities).  Observable  actions  reflecting  these  core  activities  can  then  be 
identified  and  associated  with  these  concepts  (described  below,  BBN  variables).  Lastly,  this  process  can  be 
iterated  and  the  core  concepts  linked  across  multiple  performance  events  to  permit  one  to  make  predictions 
about  a  team’s  overall  adaptive  capacity.  This  affords  the  potential  to  identify  and  subsequently  provide 
corrective/reflective  feedback  around  core  activities  of  team  adaptation  (e.g.,  situation  assessment,  planning, 
action,  monitoring)  based  on  observations  of  a  specific  performance  event  which  generalizes  to  potential  future 
events.  Such  feedback  encourages  individuals  and  teams  to  engage  in  contingency  planning,  actively  evaluate 
their  performance,  and  make  real-time  adjustments  as  needed  (i.e.,  adapt). 

To  demonstrate  proof  of  concept  and  evaluate  utility,  a  full  version  of  the  BBN  for  this  structure  was  built  using 
a  reduced  number  of  variables  (Attachment  4)  and  data  from  an  existing  dataset.  This  version  of  the  model 
spans  multiple  events  (intubation,  circulatory  support,  orthopedic  stabilization)  from  our  broader  trauma 
simulation;  Figure  6a  and  6b  provides  an  example  of  the  model  for  the  intubation  event.  Goals  for  the  model 
were  to  minimize  model  complexity  (i.e.,  number  of  modeled  relationships);  directly  map  variables/relationships 
represented  in  the  BBN  to  the  adaptability  framework  developed  in  Aim  1;  incorporate  prediction  of  medical 
task  performance  activities  into  the  model;  and  provide  a  straightforward  means  for  incorporating  feedback 
guidance  on  the  basis  of  model  predictions. 


Figure  5.  Proposed  BBN  approach 
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Figure  6a.  Sample  of  BBN  structure  applied  to  single  performance  event  (intubation) 


Area  in  grey  expanded  in  Figure  6b 


Figure  6b.  Information  gathering  and  communication  subcomponents  of  a  single  performance  event  (intubation) 


BBN  Variables:  V\le  reviewed  existing  datasets  for  candidate  variables  appropriate  for  inclusion  in  the  BBN.  This 
required  evaluating  over  100  process  variables  and  80  performance  variables.  Variables  are  considered 
appropriate  if  there  is  variability  amongst  subjects,  and  if  variables  correlate  with  overall  performance  and  process 
as  a  whole.  A  preliminary  list  of  variables  was  selected  and  underwent  subject  matter  expert  review  to  determine 
the  appropriateness  of  variables.  We  also  used  subject  matter  expert  input  to  determine  if  certain  variables  should 
be  grouped  into  composite  indicators  for  inclusion  in  the  BBN.  This  potentially  simplifies  BBN  input  during  testing 
and  refinement. 

3c.  What  opportunities  for  training  and  professional  development  has  the  project  provided? 

Subjects  enrolled  in  the  study  received  simulation-based  trauma  team  training  and  assessments.  While  the 
provision  of  training  is  not  a  major  focus  of  this  project,  trainees  were  able  to  practice  trauma  management  skills 
as  well  as  leadership  skills  under  difficult  conditions  requiring  significant  individual  and  team  adaptation. 

3d.  How  were  the  results  disseminated  to  communities  of  interest? 

Dissemination  of  our  work  from  Aims  1  a  and  1  b  is  planned  for  the  next  year.  Specifically,  we  are  preparing  two 
manuscripts,  one  describing  our  frameworks,  training  principles,  and  concepts  related  to  adaptability  and  a 
second  related  to  the  provision  of  adaptive  feedback.  Adaptive  feedback  is  a  relatively  new  concept  within 
medical  simulation  and  one  that  needs  to  be  considered  within  the  growing  literature  around  debriefing  and  the 
provision  of  performance-related  information.  We  will  also  submit  this  work  in  the  upcoming  year  to  the  Military 
Health  System  Research  Symposium,  the  Annual  Meeting  for  the  Society  for  Academic  Emergency  Medicine,  and 
the  Annual  Meeting  for  the  Human  Factors  and  Ergonomics  Society.  This  will  ensure  wide  distribution  of 
information  in  military  and  civilian  healthcare  arenas  as  well  as  within  the  training  and  human  factors  community. 


3e.  What  do  you  plan  to  do  during  the  next  reporting  period  to  accomplish  the  goals? 

Aim  la.  Develop  a  team  training  design  architecture  to  support  simulation-based  training/assessment 
systems  capable  of  developing  adaptive  expertise  in  health  care  teams 
Aim  1b.  Develop  evidence-based  guidelines  and  recommendations  for  the  development  of  embedded, 
adaptive  feedback  and  performance  assessments 

Work  for  Aims  1  a  and  1  b  are  completed. 

Deliverable  1.  Health  Care  Team  Training  Design  Architecture.  A  unified,  evidence-based  conceptual  framework 
of  health  care  team  training  effectiveness  that  identifies  critical  variables  -  individual  and  team 
factors,  training  design  elements,  and  training  implementation  methods  -  that  can  be  leveraged  to 
improve  team  adaptive  expertise  and  performance  through  robust  simulation-based  training 
systems 

Deliverable  2.  Embedded,  Adaptive  Guidance:  Guidelines  and  Recommendations.  Clear  guidelines  and 
recommendations  for  the  design,  development,  and  implementation  of  embedded,  adaptive 
guidance  to  optimize  team  adaptability  and  team  performance 

Concepts  and  principles  identified  through  Aims  la  and  1b  will  be  disseminated  in  manuscripts  during  the  NCE 
year.  Planned  targets  include  Academic  Medicine  and  Academic  Emergency  Medicine.  We  will  also  submit  this 
work  in  the  upcoming  year  to  the  Military  Health  System  Research  Symposium,  the  Annual  Meeting  for  the 
Society  for  Academic  Emergency  Medicine,  and  the  Annual  Meeting  for  the  Human  Factors  and  Ergonomics 
Society.  This  will  ensure  wide  distribution  of  information  in  military  and  civilian  healthcare  arenas  as  well  as  within 
the  training  and  human  factors  community. 


Aim  2.  Develop  and  refine  a  predictive  model  of  trauma  team  performance  and  outcomes  for  use  in  an 
adaptive  guidance/feedback  system 

Trauma  Simulations  and  Performance  Coding:  The  purpose  of  conducting  trauma  team  simulations  is  to 
provide  baseline  data  for  the  design  of  the  BBN.  To  date  we  have  conducted  all  of  the  proposed  simulations.  We 
will  continue  to  perform  simulations  involving  military  providers  to  ensure  adequate  representation  but  have 
collected  the  data  necessary  to  move  ahead  with  the  BBN  design  and  testing.  New  simulations  will  be  coded  on 
an  ongoing  basis  throughout  the  next  project  year. 

Generation  of  Initial  Probability  Tables  for  BBN  Team  Assessment  System:  The  computational  “engine”  and 
predictive  validity  of  a  BBN  relies  on  the  presence  of  well-informed  conditional  probability  tables  (CPTs).  A  CPT 
exists  for  every  node  in  a  BBN  and  reflects  the  probability  that  a  particular  state  for  a  particular  node  will  be 
observed  given  the  state  of  all  its  parent  nodes  (e.g.,  p(Chest  Compression  Quality  =  High  |  Assign  a  Team 
Leader  =  No),  etc.).  In  this  sense,  CPTs  represent  the  degree  of  interdependency  (i.e.,  correlation)  that  exists 
between  variables  that  share  a  directed  arc.  To  compute  the  CPTs  for  the  candidate  networks,  the  investigators 
will  utilize  their  existing  dataset  to  “train”  a  set  of  initial  conditional  probabilities  for  the  modeled  variables.  This 
process  will  entail  several  steps.  First,  data  must  be  transformed  into  an  appropriate  categorical  structure  that  can 
be  interpreted  by  a  BBN.  Next,  different  learning  algorithms  will  be  explored  (i.e.,  counting,  expectation- 
maximization,  gradient  descent)  in  an  attempt  to  produce  the  “maximum  likelihood  BBN,”  or  the  set  of  CPTs  that 
is  most  likely  given  the  observed  data.  The  fit  of  the  algorithms  will  be  assessed  using  standard  model  evaluation 
techniques  (e.g.,  confusion  matrix,  times  surprised,  etc.);  additionally,  these  metrics  will  be  used  to  compare 
candidate  BBNs  to  identify  the  best  fitting  model.  Finally,  in  instances  where  data  is  unavailable  or  insufficient  to 
generate  a  suitable  CPT,  existing  empirical  literature  (i.e.,  meta-analyses)  and/or  subject  matter  experts  will  be 
relied  upon  to  generate  the  nature  of  the  conditional  dependence.  The  result  of  this  step  will  be  development  of 
the  best  fitting,  functional  prototype  BBN  team  assessment  system  based  on  existing  data. 

BBN  Team  Assessment  System  Calibration:  A  potential  concern  with  using  only  a  single  sample  to  construct  a 
BBN  is  that  the  model  and  its  accompanying  CPTs  may  be  “overfit”  and  fail  to  generalize  beyond  the  training 
data.  Thus,  in  the  final  step  of  development,  the  performance  of  the  BBN  team  assessment  system  will  be 
evaluated  and  recalibrated  using  the  new  data  collected  through  coding  of  simulations.  A  similar  approach  to 
evaluating  model  fit  as  described  above  will  be  implemented  to  examine  the  adequacy  of  the  BBN’s  predictions  in 
the  new  data.  To  the  extent  that  misfits  among  particular  nodes  or  relationships  are  identified,  the  investigators 
will  rely  on  subject  matter  experts  and  empirical  evidence  from  the  literature  to  identify  whether  and/or  how  to 
adapt  the  BBN  (adjust  CPTs,  specify  new  nodes/variables,  revise  causal  pathways).  Irrespective  of  fit,  the  new 
data  can  be  used  to  improve  the  precision  of  the  BBN  assessment  model  through  added  observations.  The 
results  of  this  step  will  thus  be  improvement  and  calibration  of  the  prototype  BBN  team  assessment  system. 


4.  Impact  for  Project  Year  2  Work 


4a.  What  was  the  impact  on  the  development  of  the  principal  discipline(s)  of  the  project? 

Our  work  will  improve  training,  maximize  healthcare  provider  performance,  and  minimize  morbidity  for  our 
injured  service  men  and  women.  Once  disseminated,  the  work  from  project  year  one  will  provide  military  and 
civilian  healthcare  providers  and  educators  with  clear  guidelines  for  the  development  of  training  that  builds 
adaptive  capacity.  Specifically,  we  provide  developmentally  appropriate  training  targets  for  individuals  and  teams. 
We  identify  what  training  content  and  delivery  method  is  most  appropriate  for  developing  adaptive  behaviors 
around  certain  types  of  tasks.  We  recognize  that  frontline  medics  adapt  to  different  situations  than  physicians  in 
specialty  clinics  and  our  guidelines  account  for  these  differences.  We  aim  to  provide  a  clear,  easily  applied 
method  to  help  educators  and  trainers  make  decisions  regarding  training  development  and  implementation.  Our 
work  will  facilitate  the  development  of  longitudinal  curricula  across  multiple  specialties  and  disciplines  by  providing 
clear  training  targets  for  individuals  and  teams  at  all  levels  of  performance. 

The  guidelines  and  principles  for  adaptive  feedback  introduce  a  new  and  important  concept  to  healthcare.  The 
provision  of  “feedback”  and  “debriefing”  in  experiential  training  has  been  identified  as  critical  to  learning. 

However,  the  role  for  adaptive  feedback  in  the  development  of  highly  adaptive  teams  has  not  been  described.  We 
will  disseminate  our  review  of  the  topic  along  with  specific  recommendations  for  implementation  within  simulation- 
based  training.  Along  with  the  work  to  be  performed  in  Aim  2,  this  information  will  provide  the  foundation  for  the 
development  of  simulation-based  training  with  automated,  adaptive  feedback. 

The  development  of  a  predictive  model  of  trauma  team  performance  is  underway.  When  completed,  this  work 
will  apply  a  novel  approach  to  the  provision  of  adaptive  guidance  and  feedback  during  team  performance.  This 
will  not  only  advance  our  understanding  of  successful  team  performance,  but  will  also  inform  educators  about  how 
the  delivery  of  feedback  and  guidance  can  impact  adaptive  performance. 

4b.  What  was  the  impact  on  other  disciplines 

Our  work  has  impact  beyond  healthcare.  We  highlight  the  challenges  associated  with  training  and  evaluating 
performance  in  complex  environments.  This  information  is  useful  in  human  factors  and  organizational  psychology, 
where  teamwork  has  often  been  considered  a  static  construct,  rather  than  a  dynamic  entity  where  teams  learn, 
adapt,  and  react  to  continuous  changes  in  the  task,  environment,  and  team.  Our  framework  highlights  how 
important  it  is  to  consider  characteristics  of  the  task(s)  necessitating  adaptation  when  developing  training 
programs.  This  work  provides  a  foundation  to  build  more  comprehensive  training  that  goes  beyond 
TeamSTEPPS-type  training  to  impact  complex  teams  performing  in  highly  dynamic,  potentially  dangerous 
situations.  Additionally,  the  application  of  BBNs  as  an  analytical  framework  has  primarily  been  restricted  to 
problem  domains  within  engineering  and  ecology.  The  use  of  these  techniques  for  modeling  individual  and  team 
behavior  as  well  as  for  guiding  the  delivery  of  feedback  is  both  novel  and  highly  generalizable.  With  respect  to 
healthcare  applications,  the  application  of  BBNs  we  have  pursued  to  model  team  performance  can  be  extended  to 
all  disciplines  within  healthcare,  including  forward  military  units,  ambulatory  care  centers,  and  long-term 
rehabilitation  units.  The  use  of  adaptive  guidance  can  be  incorporated  into  automated,  online  training  as  well  as 
mannequin-based  simulation  curricula. 

4c.  What  was  the  impact  on  technology  transfer? 

Nothing  to  report 

4d.  What  was  the  impact  on  society  beyond  science  and  technology? 

Failure  to  adapt  to  rapidly  changing  conditions  is  a  primary  cause  of  medical  error.  In  military  settings,  such 
failures  can  also  lead  to  significant  harm  to  providers.  Our  work  has  a  significant  impact  on  patient  safety, 
decreasing  soldier  morbidity  and  mortality,  and  on  patient  satisfaction.  Simulation  is  a  key  modality  leveraged  by 
the  military  to  advance  expertise  and  ensure  that  soldiers  receive  the  highest  level  of  clinical  care.  Significant 
human  and  technological  resources  are  dedicated  to  developing  and  implementing  rigorously  tested,  high-quality 
simulation-based  curricula.  Clear  guidelines  and  a  training  framework  focused  on  developing  adaptive  capacity 
did  not  exist.  We  fill  this  gap  and,  in  doing  so,  provide  an  important  mechanism  to  support  the  development  and 
implementation  of  highly  effective  individual  and  team-level  healthcare  training. 


5.  Changes  /  Problems 


5a.  Changes  in  approach  and  reasons  for  change 

There  are  no  anticipated  changes  in  approach. 

5b.  Actual  or  anticipated  problems  or  delays  and  actions  or  plans  to  resolve  them 

PI  Relocation:  The  PI,  Rosemarie  Fernandez  is  relocating  to  the  University  of  Florida-  Jacksonville  at  the 
beginning  of  January.  This  move  will  not  change  the  scope  of  the  work  to  be  completed  nor  result  in  any  changes 
to  the  budget.  We  fully  anticipate  completing  the  work  proposed  for  the  NCE  year. 

Team  engagement:  The  team  will  continue  to  meet  weekly  via  GoToMeeting.  Dr.  Fernandez  will  use  funds  set 
aside  for  travel  to  have  at  least  one  in-person  meeting  in  Seattle  to  facilitate  the  final  components  of  the  project. 
Budget:  The  team  will  continue  to  execute  the  proposed  work.  There  are  adequate  funds  available  and  both 
institutions  (University  of  Washington  and  University  of  Florida  -  Jacksonville)  agree  to  the  terms  of  the  new 
budget.  We  therefore  do  not  anticipate  any  issues. 

Simulation  delay:  Scheduling  the  simulations  was  slightly  delayed  due  to  the  residents’  (subject)  clinical 
schedules.  Simulations  are  complete.  To  ensure  adequate  military  representation,  we  continued  executing 
simulations  through  the  beginning  of  the  NCE  year  to  obtain  the  maximal  amount  of  military  subject  data.  To 
augment  simulation  performance  and  coding,  we’ve  added  a  co-investigator  (M.  Vrablik)  to  the  research  team. 

We  are  able  to  do  this  while  staying  within  the  proposed  budget.  We  do  not  anticipate  any  additional  delays. 

Subject  Matter  Expert  Input:  We  elected  to  get  further  input  from  our  subject  matter  experts  for  the  BBN 
variables  and  relationships.  This  pushed  the  BBN  testing  and  calibration  to  the  NCE  year.  This  work  is  underway 
and  does  not  reflect  a  change  in  the  approach.  The  work  for  Aim  2  will  still  be  completed  within  the  proposed 
budget. 

5c.  Changes  that  had  a  significant  impact  on  expenditures 

The  project  is  currently  on  budget.  Delays  in  hiring  the  research  assistant  and  delay  in  starting  simulations  have 
shifted  some  of  the  costs  to  the  NCE  year.  Subcontract  costs  are  encumbered  now  for  years  1  and  2.  The  slight 
delays  described  above  do  not  impact  the  budget,  and  we  fully  anticipate  completing  the  project  within  the 
proposed  budget. 

5d.  Significant  changes  in  use  or  care  of  human  subjects,  vertebrate  animals,  biohazards,  and/or  select 
agents 

None.  While  we  increased  our  simulation  subjects  to  22  teams,  this  protocol  is  exempt  and  no  further  action  is 
needed. 


6.  Products 


6a.  Publications,  conference  papers,  and  presentations 

1.  Fernandez  R,  Rosenman  ER;  Santoro  J,  Pacic  E,  Golden  SJ,  Brolliar  SM,  Chao  GT,  Grand  JA,  Kozlowski 
SWJ.  A  multicenter,  observational  study  of  teamwork,  team  cognition,  and  leadership.  2016  Military  Health 
System  Research  Symposium ,  Orlando,  FL. 

2.  Rosenman  ED,  Dixon  AJ,  Webb  JM,  Brolliar  SM,  Golden  SJ,  Jones  KA,  Shah  S,  Grand  JA,  Kozlowski 
SWJ,  Chao  GT,  Fernandez  R.  A  simulation-based  approach  to  measuring  team  situational  awareness  in 
emergency  medicine:  A  multicenter,  observational  study.  Acad  Emer  Med  (doi:  1 0.1 111/acem.l 3257). 

6b.  Website  or  other  Internet  sites 

None 

6c.  Technologies  or  techniques 

None 

6d.  Inventions,  patent  applications,  and/or  licenses 

None 

6e.  Other  products 

None 


7.  Participants  &  Other  Collaborating  Organizations 

7a.  What  individuals  have  worked  on  the  project? 

Name: 

Project  role: 
era  Commons  ID: 

Nearest  person  month  worked: 
Contribution  to  project: 

Rosemarie  Fernandez,  MD 

Principal  Investigator 
av9546 

2.4  cal.  Months  (0.2  FTE) 

Worked  with  ER  to  review  actual  trauma  resuscitations  and  identify 
missing  components  of  the  conceptual  framework.  Worked  with  JG,  ER, 
GC  to  address  issues  raised  during  IPR.  Recruited  subjects  for 
simulation  and  executed  simulations.  Worked  with  ER,  JG  to  help  define 
the  BBN  structure  and  develop  a  prototype. 

Name: 

Project  role: 
era  Commons  ID: 

Nearest  person  month  worked: 
Contribution  to  project: 

James  Grand,  PhD 

Co-Principal  Investigator 

Grandjam 

3  cal.  Months  (0.25  FTE) 

Worked  with  GC  to  conduct  team  science  component  of  literature  review. 
Worked  with  RF,  ER,  GC  to  address  issues  raised  during  IPR.  Led  efforts 
in  the  development  of  the  BBN  structure  and  develop  a  prototype. 

Name: 

Project  role: 

Nearest  person  month  worked: 
Contribution  to  project: 

Elizabeth  Rosenman,  MD 

Co-Investigator 

1.2  cal.  Months  (0.1  FTE) 

Worked  with  RF  to  review  actual  trauma  resuscitations  and  identify 
missing  components  of  the  conceptual  framework.  Worked  with  JG,  RF, 
GC  to  address  issues  raised  during  IPR.  Recruited  subjects  for 
simulation  and  executed  simulations.  Worked  with  RF,  JG  to  help  define 
the  BBN  structure  and  develop  a  prototype. 

Name: 

Project  role: 

Nearest  person  month  worked: 
Contribution  to  project: 

Georgia  Chao,  PhD 

Co-Investigator 

2  cal.  Months  (0.19  FTE) 

Worked  with  JG  to  conduct  team  science  component  of  literature  review. 
Worked  with  RF,  ER,  JG  to  modify  conceptual  framework.  Identified  team 
science-related  training  principles  and  recommendations. 

Name: 

Project  role: 

Nearest  person  month  worked: 
Contribution  to  project: 

CPT.  Lindsay  K.  Grubish,  DO 

Military  investigator 

0.6  cal  months  (0.05  FTE) 

Provides  military  subjects  and  acts  as  subject  matter  expert  for  military 
medicine. 

Name: 

Project  role: 

Nearest  person  month  worked: 
Contribution  to  project: 

Ly  Huynh,  BA 

Research  assistant 

6  cal.  Months  (0.5  FTE)  -  STARTED  11/2016 

Replaced  K.  Jackson.  Coordinated  subject  recruitment,  worked  with 
simulation  center  to  schedule  and  execute  simulations.  Worked  on 
developing  comprehensive  database  and  secure  transfer  of  data. 

Name: 

Project  role: 

Nearest  person  month  worked: 
Contribution  to  project: 

Benjamin  Levine,  BA 

Graduate  student  research  assistant 

6  cal.  Months  (0.5  FTE) 

Worked  with  team  to  develop  BBN  approach  and  development  of 
prototype  BBN  with  relationships. 

Name:  Jessica  Santoro,  MA 

Project  role:  Graduate  student  research  assistant 

Nearest  person  month  worked:  1  cal.  Months  (will  be  at  0.5  FTE  for  upcoming  project  year) 

Contribution  to  project:  Assisted  with  development  of  adaptive  feedback  principles,  performed 

related  literature  review,  linked  principles  with  TeamSTEPPS  principles. 

7b.  Has  there  been  a  change  in  the  active  other  support  of  the  PD/PIs  or  senior/key  personnel  since  the 
last  reporting  period? 

We  added  CPT.  Lindsay  K.  Grubish,  DO  to  the  project,  as  Dr.  Jay  Baker  took  a  new  position  and  is  no  longer  able 
to  assist  with  the  project.  CPT.  Grubish  is  a  staff  physician  in  the  Department  of  Emergency  Medicine  at  MAMC. 
She  has  experience  using  simulation  to  assess  performance  in  military  medical  providers  under  stress.  She  will 
be  responsible  for  providing  subject  matter  expertise  and  recruitment  of  military  residents  for  the  simulations. 
Attached  is  her  Biosketch  (Attachment  5). 

As  noted  above,  we  added  Dr.  Marie  Vrablik  to  the  project  to  assist  with  coding  of  videos  (Biosketch  Attachment 
6). 


7c.  What  other  organizations  were  involved  as  partners? 


University  of  Maryland 

Department  of  Psychology 
College  Park,  Maryland 

The  Co-PI,  Dr.  Grand,  and  a  graduate  student,  Mr. 
Benjamin  Levine,  are  both  supported  at  the  University 
of  Maryland.  There,  they  have  office  space,  computer 
access,  and  support  for  virtual  meetings  with  the 
research  team. 

Madigan  Army  Medical  Center 

9040  Jackson  Ave. 

Tacoma,  WA  98431 
Co-1:  CAPT.  L.  Grubish 

CAPT.  Grubish  will  assist  with  subject  matter  expert 
queries  and  will  also  assist  with  simulations  and 
performance  coding. 

8.  Special  Reporting  Requirements 


Eli  Broad  College  of  Business  /  Michigan  State 
University 

East  Lansing,  Michigan 

Dr.  Chao  (collaborator)  and  a  graduate  student,  Ms. 
Jessica  Santoro,  are  both  supported  at  Michigan 
State  University.  There,  they  have  office  space, 
computer  access,  and  support  for  virtual  meetings 
with  the  research  team. 


8a.  Collaborative  Awards 

N/A 


8b.  Quad  Chart 

Please  see  Attachment  7  for  updated  Quad  Chart. 
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ATTACHMENTS 

Training  principles  to  target  adaptive  processes  at  different 
levels 

Training  principles  related  to  task  type  and  complexity 
Principles  of  providing  adaptive  feedback 
Variables  for  the  BBN  model 

L.  Grubish  CV 

M.  Vrablik  CV 
Project  QUAD  Chart 


Attachment  1:  Training  principles  to  target  adaptive  processes  at  different  levels. 


Principle  and 

Applicable  Level(s) 

Rationale 

Simulation  application 

TeamSTEPPS  Associations 

Use  pre-training 
materials  to  provide 
appropriate  orientation 
to  trainees. 

(Individual  Level) 

Pre-training  materials  presented  at  the  start 
of  training  provide  an  initial  organizing 
structure  of  the  subject  matter  discussed  in 
training.  Pre-training  materials  provide 
conceptual  information,  help  to  build 
connections  between  similar  ideas,  and 
delineate  different  concepts  from  one 
another.  Trainees  who  use  or  begin  to 
develop  their  own  pre-training  materials  are 
more  likely  to  adaptively  transfer  knowledge 
and  skills. 

•  Inform  trainees  about  training  focus. 

This  does  not  necessarily  mean 
informing  them  of  key  critical  content 
planned  for  simulations;  rather,  tell 
trainees  they  will  be  focusing  on  team 
(or  individual)  skills 

•  Suggest  that  trainees  consider 
personal  strengths  and  weaknesses 
prior  to  coming  to  training. 

•  No  associations 

Promote  trainees  to 
have  a  learning  goal 
orientation  during 
training. 

(Individual  and  Team 
Level) 

Training  design  that  promotes  a  learning  goal 
orientation  (e.g.,  a  focus  on  self-improvement 
and  task  mastery  in  achievement  situations) 
has  been  linked  to  positive  training 
outcomes,  such  as  goal  setting,  self- 
regulatory  activities,  learning,  and 
performance.  This  is  in  stark  contrast  to 
promoting  a  performance  goal  orientation 
(e.g.,  a  focus  on  demonstrating  ability  to 
others  in  achievement  situations)  which  has 
been  shown  to  negatively  relate  to  goal 
striving  processes  and  performance. 

•  Promote  a  learning  goal  orientation  by 
encouraging  trainees  to  set  goals  about 
achieving  learning  objectives  and 
acquiring  relevant  knowledge  and 
skills. 

•  Establish  psychological  safety 

•  Psychological  safety  is  about  being 
able  to  take  interpersonal  risks  on  a 
team.  The  concept  of  psychological 
safety  has  similarity  to  TeamSTEPPS' 
mutual  trust  dimensions  of  “advocacy 
and  assertion”  and  “two-challenge 
rule”.  These  two  dimensions  discuss 
the  role  of  speaking  up  about  decisions 
being  made  within  the  team.  The 
advocacy  and  assertion  piece  asks 
team  members  to  voice  new  viewpoints 
that  clash  with  the  leader’s  viewpoint. 
They  are  asked  to  assert  themselves 
firmly  and  respectfully.  The  two- 
challenge  rule  piece  describes  that  if 
an  initial  assertion  goes  unanswered, 
the  team  member  should  assert  at 
least  twice  to  ensure  their  viewpoint  is 
heard.  (Ferguson,  p.  123) 

Trainees  should  be 
provided  with  higher- 
level  coordination 
strategy  instruction  later 
in  training  once 
appropriate  foundational 
knowledge  has  been 
developed. 

(Individual) 

The  KSAs  required  to  effectively  engage  in 
individual  and  team  adaptation  are  advanced 
learning  outcomes.  Without  achieving 
proficiency  in  the  basic  and  procedural 
knowledge  necessary  to  carry  out  core 
task/job  requirements  in  a  domain,  efforts  to 
improve  the  adaptation  process  will  be  less 
effective. 

•  Assess  individuals  for  team-based 
simulation  "readiness" 

•  Use  low  fidelity  non-clinical  simulations 
to  begin  building  team  skills  while 
individuals  are  still  developing  clinical 
knowledge. 

•  At  this  stage,  interdisciplinary  training  is 
not  important;  however  institutions 
should  ensure  consistency  of 
curriculum  across 
professions/units/schools 

•  No  associations 

Adopt  a  Crawl-Walk- 

Run  approach  to 
training  design.  Training 
material  should  be 
structured  so  that 
instruction  proceeds 
from  general  to  detailed, 
specific  to  complex. 

(Individual  and  Team 
Level) 

Successful  team  adaptation  requires 
integrating,  coordinating,  and  regulating  a 
variety  of  different  KSAs,  resources,  and 
members.  Developing  the  capacities  to 
manage  these  processes  should  be  built 
around  a  Crawl-Walk-Run  curriculum  model 
to  allow  learners  to  first  achieve  basic 
competencies  and  then  practice/engage  in 
more  complex  applications.  Note  that  this 
also  applies  to  actively  training  members  as 
part  of  intact  teams  —  team-based  training 
designed  to  enhance  adaptability  is  a 
complex  environment  and  should  be 
postponed  until  learners  have  engaged  in 
more  foundational  training  exercises. 

•  Team-based  simulations  should  initially 
use  basic  clinical  scenarios  rather  than 
unusual  or  highly  complex  situations. 
Once  basic  team  skills  have  transferred 
from  "non-clinical"  simulations  (above) 
to  straightforward  clinical  issues,  more 
complex  team  and  environmental 
issues  can  be  added. 

•  Use  EBAT  to  create  a  simulation 
experience  where  modules  can  be 
added  to  model  more  complexity  as 
well  as  to  target  specific  team  skills. 

•  No  associations 

Trainees  learning  a 
complex  task  should  be 
encouraged  to  monitor 
rate  of  learning  progress 
rather  than  just  learning 
performance. 

(Individual  Level) 

Training  that  emphasizes  learning 
trajectories,  development,  and  velocity  is 
more  likely  to  minimize  goal  abandonment, 
promote  self-efficacy,  and  encourage 
trainees  to  view  training  as  "learning"  rather 
than  "evaluation."  Additionally,  emphasizing 
"future-focused"  cognitive  appraisals  (i.e., 
focusing  on  how  learning 
outcomes/capabilities  are  evolving) 
reinforces  the  cognitive  appraisal  frames 
critical  to  team  adaptation. 

•  During  pre-brief,  make  it  clear  to 
learners  that  there  may  be  no  "right 
answer". 

•  Establish  a  learning  environment  that 
supports  psychological  safety. 

•  If  using  a  modular  EBAT  approach, 
consider  guiding  teams  to  recognize 
how  similar  problems  were  addressed 
in  the  past  so  they  can  monitor  their 
progress. 

•  Psychological  safety  is  about  being 
able  to  take  interpersonal  risks  on  a 
team.  The  concept  of  psychological 
safety  has  similarity  to  TeamSTEPPS' 
mutual  trust  dimensions  of  “advocacy 
and  assertion”  and  “two-challenge 
rule”.  These  two  dimensions  discuss 
the  role  of  speaking  up  about  decisions 
being  made  within  the  team.  The 
advocacy  and  assertion  piece  asks 
team  members  to  voice  new  viewpoints 
that  clash  with  the  leader’s  viewpoint. 
They  are  asked  to  assert  themselves 
firmly  and  respectfully.  The  two- 
challenge  rule  piece  describes  that  if 
an  initial  assertion  goes  unanswered, 
the  team  member  should  assert  at 
least  twice  to  ensure  their  viewpoint  is 
heard.  (Ferguson,  p.  123) 

Trainees  learning 
complex  tasks  should  be 
provided  with  proximal 
subgoals  that  break  the 
task  into  smaller  parts. 
(Individual  and  Team 
Level) 

Team  adaptation  is  a  process  characterized 
by  an  ongoing  cycle  of  situation  assessment 
and  team/task  management.  The  KSAs 
which  underlie  successfully  execution  of 
these  stages  can  be  developed  through 
"part-learning"  and  by  breaking  the 
adaptation  process  into  meaningful  chunks. 
This  approach  is  more  likely  to  increase 
learner  self-efficacy  and  persistence,  and 
allow  practice  opportunities  &  feedback  to  be 
tailored  towards  more  focused  learning 
objectives. 

•  Break  down  adaptive  behaviors  into 
clear  activities  that  can  be  practiced  in 
isolation.  If  necessary,  remove  learners 
from  the  clinical  setting  to  work  on  key 
activities  prior  to  re-entering  a  high- 
fidelity  simulation. 

•  No  associations 

Trainees  presented  with 
extremely  difficult 
problems  that  appear 
unsolvable  should  be 
assisted  in  making  some 
consistent  progress 
during  training. 

(Individual  Level) 

The  structure  of  the  training  environment  and 
practice  opportunities  for  team  adaptability 
should  not  be  "sink  or  swim"  (esp.  during 
initial  stages  of  practice).  Feedback  and 
direction  that  actively  guides  teams  through 
how  to  think  through  a  complex  task  and 
make  decisions  about  resources  is  a  critical 
foundation  of  team  adaptability  training. 
Providing  guidance  that  prompts  teams  to 
explore  options  for  task  completion  during 
training  helps  to  avoid  discouragement, 
anxiety,  and  abandonment  of  effort. 

•  Use  triggers  and  backup  triggers  during 
simulations  to  allow  learners  to  attempt 
the  behavior  and,  if  unsuccessful, 
observe  an  "expert"  (confederate) 
execute  the  behavior  with  success. 

•  Junior  learners  that  may  lack  clinical 
knowledge  should  be  encouraged  to 
seek  assistance  for  help  at  any  time. 
Using  confederates  as  "mentors"  can 
not  only  assist  learners  through  difficult 
tasks  but  also  will  build  comfort  with 
seeking  help  from  other  team  members 
and  those  outside  the  team. 

•  In  performance  episodes,  task 
assistance  occurs  through 
TeamSTEPPS'  mutual  support  tool 
when  “team  members  foster  a  climate 
where  it  is  expected  that  assistance  will 
be  actively  sought  and  offered” 
(Ferguson,  p.  123) 

Variability  in  practice 
trials  should  be  provided 
during  training  to 
maximize  retention  & 
transfer. 

(Individual  and  Team 
Level) 

Whereas  early  stages  of  training  are 
enhanced  by  repetition  and  rehearsal  (i.e. , 
developing  declarative  &  procedural 
knowledge),  advanced  stages  of  training  are 
enhanced  by  exposing  trainees  to  as  diverse 
an  array  of  scenarios  in  which  to  apply  their 
KSAs  as  possible.  It  is  particularly  critical  to 
expose  trainees  to  situations  where 
previously  learned,  frequently  used,  and/or 
typically  reliable  courses  of  action  are 
ineffective.  Providing  variability  in  practice 
trials  promotes  the  development  of  broader 
associative  knowledge  structures  and 
contingency-based  thinking. 

•  Use  EBAT  to  build  simulations  that 
contain  appropriate  task  complexity 

•  Shorten  intervals  between  prompts  to 
increase  time  pressures  as 
appropriate. 

•  Use  confederates  to  add  interpersonal 
challenges. 

•  Build  in  environmental  challenges  (e.g., 
additional  patients,  equipment  failure) 
to  increase  complexity 

•  No  associations 

Training  should  be 
permissive  of,  embrace, 
and  even  encourage 
errors  made  by  learners 
during  training. 

(Individual  and  Team 
Level) 

Errors  are  an  inevitable  component  of  real- 
world  performance.  Errorless  training  leads 
to  effective  training  performance,  but  is  often 
related  to  poor  training  transfer.  Although 
errors  during  training  should  be  brought  to 
learners’  attention,  learning  that  is  focused 
on  error  management  as  opposed  to  error 
prevention  is  more  successful.  Framing 
training  as  an  opportunity  to  make  and  learn 
from  errors  encourages  trainees  to  develop 
problem-solving  or  hypothesis-testing  skills 
and  strategies  for  managing  affective 
responses  (e.g.,  frustration  and  anxiety). 

•  Use  confederates  to  “force”  errors 
during  simulations.  This  requires 
considerable  expertise  in  debriefing  to 
ensure  learners  do  not  feel  “tricked”. 
Appropriate  pre-briefing  and 
establishment  of  a  learning 
environment  can  help.  Be  sure  that 
“errors”  meet  a  minimum  level  of 
psychological  fidelity  for  learners. 

TeamSTEPPS  takes  a  slightly  different 
view  of  errors  and  does  not  specifically 
address  the  use  of  errors  in  training. 

•  TeamSTEPPS  argues  that 
performance  should  be  error  free,  but 
does  not  talk  about  the  conditions  for 
training.  They  advocate  for  situation 
monitoring  whereby  team  members 
monitor  the  actions  of  other  team 
members  for  the  purpose  of  reducing 
and  avoiding  errors.  (Ferguson,  p.  123) 

•  TeamSTEPPS  would  advocate  for 
team  members  to  monitor  the 
environment  to  look  for  these  errors  so 
that  they  are  caught  “quickly  and 
easily”.  They  encourage  for  team 
members  to  watch  each  other's  backs. 

Incorporate  lessons  on 
how  to  alter  coordination 
strategies  in  training. 
(Team  Level) 

When  task  demands  are  low,  trainees  should 
learn  to  discuss  possible  problems  that  could 
arise  later  in  the  task.  By  discussing  their 
coordination  strategies  during  this  period, 
they  will  likely  reduce  the  amount  of 
communication  necessary  to  achieve 
successful  team  performance  later  and  allow 
them  to  be  adaptive  when  novel  problems 
arise  in  the  environment. 

•  Encourage  learners  to  develop 
contingency  plans 

•  Discuss  team  member  understanding 
and  mental  model  development  during 
debriefing  to  help  reinforce  the 
importance  of  discussing  and  practicing 
team  coordination 

•  TeamSTEPPS  offers  the  leadership 
tool  called  the  “brief”,  which  is  a  “short 
session  prior  to  start  to  share  the  plan, 
discuss  team  formation,  assign  roles 
and  responsibilities,  establish 
expectations  and  climate,  anticipate 
outcomes  and  likely  contingencies”. 
(Pocket  Guide,  p.  16) 

•  Use  of  the  term  “mental  model”  is 
consistent  with  TeamSTEPPS 
language.  A  situation  monitoring  tool  is 
the  shared  mental  model,  which 
Ferguson  defines  as  “the  perception  of, 
understanding  of,  or  knowledge  about 
a  situation  or  process  that  is  share 
among  team  members  through 
communication.  Having  team  members 
on  the  same  page  is  the  desired  team 
outcome.”  (p.  123) 

•  Debriefing  in  TeamSTEPPS  is  referred 
to  as  “Process  improvement  -  Debrief” 
where  an  after-action  review  is  used  “to 
provide  feedback  and  improve  team 
performance”.  (Ferguson,  p.  123) 

Integrate  metacognitive 
prompts  into  training. 
(Individual  Level) 

Metacognition  is  the  process  of  actively 
reflecting  on  one's  thought  processes. 
Encouraging  metacognitive  activity  during 
training  can  help  learners  identify  and  focus 
on  the  goals,  assumptions,  and  strategies 
guiding  their  decision-making  and  task 
performance.  This  is  especially  important  for 
less  experienced  trainees  learning  to  perform 
in  complex  and  dynamic  environments  and 
who  may  struggle  with  such  “big  picture” 
thinking. 

•  Employ  “think  aloud”  protocols  during 
simulation-based  training  in  which  the 
trainee  verbalizes  their  thought  process 
during  practice 

•  Build  in  opportunities  for  more  frequent 
huddles  during  simulation-based 
training  in  which  the  trainee  is 
prompted  to  explicitly  discuss  their 
rationale  for  previous  decisions  and 
considerations  for  future  plans. 

•  TeamSTEPPS  encourages  talking  out 
loud  even  during  performance 
episodes.  It's  referred  to  as  a  “call-out” 
where  team  members  are  informed 
simultaneously.  While  this  isn't  a 
“thinking”  procedure,  the  two  methods 
are  similar  in  the  way  that  they  are 
performed. 

Attachment  2.  Identifying  Task  Complexity  and  Associated  Best  Practice  Training  Principles 


Adapting  to  changes  in  Component 
complexity 

Changes  in  number  and/or  difficulty  of  tasks 

Adapting  to  changes  in  Coordinative  complexity 

Changes  in  sequencing,  prioritization,  & 
interdependence  among  tasks 

Adapting  to  changes  in  Dynamic  complexity 

Volatility  in  component  &  coordinative  complexity 
within  a  task 

Principle 

Rationale 

Principle 

Rationale 

Principle 

Rationale 

Trainees  should  not 
be  provided 
complex 
coordinative 
instruction  until  later 
in  training 

Emphasizing  breaking 
down  tasks  into  subtasks 
and  how  to  complete  small 
numbers  of  simple, 
manageable  tasks  during 
early  knowledge/skill 
acquisition  promotes  self- 
efficacy  and  draws  focus 
away  from  premature 
comparative  &  normative 
evaluations 

Trainees  should  not 
be  provided 
complex, 
coordinative 
instruction  until  later 
in  training 

Shifting  training  towards 
prioritization,  how  to  develop 
contingencies*,  and  managing 
distal  vs.  proximal  goals  once 
trainees  have  achieved 
proficiency  in  basic  knowledge 
and  skill  promotes  mastery 
learning  and  promotes  "big 
picture"  thinking 

Trainees  should  not 
be  provided 
complex, 
coordinative 
instruction  until  later 
in  training 

Shifting  training  towards 
recognizing  when  change  is 
needed  and  when/how  to 
implement  contingencies* 
focuses  trainees  appropriately 
on  normative  expectations  and 
being  proactive. 

Training  material 
should  be 
structured  so  that 
instruction  proceeds 
from  general  to 
detailed,  specific  to 
complex 

Training  experiences 
should  support  trainees 
learning  to  deal  with 
few/simple  tasks  — > 
more/simple  tasks  --> 
few/difficult  tasks  — > 
more/difficult  tasks.  This 
enables  training/feedback 
to  focus  on  quantity  vs. 
complexity  of  tasks,  which 
pose  different 
considerations 

Training  material 
should  be 
structured  so  that 
instruction  proceeds 
from  general  to 
detailed,  specific  to 
complex 

Training  experiences  should 
support  trainees  learn  to  deal 
with  few/simple  tasks  --> 
more/simple  tasks  — > 
few/difficult  tasks  — > 
more/difficult  tasks.  This 
enables  training/feedback  to 
focus  on  quantity  vs. 
complexity  of  tasks,  which 
pose  different  considerations 

Training  material 
should  be 
structured  so  that 
instruction  proceeds 
from  general  to 
detailed,  specific  to 
complex 

Training  that  allows  practice 
shifting  from  few/simple  tasks 
to  more/complex  tasks  within 
the  learning  environment 
allows  learners  to  practice 
situation  assessment  and  task 
regulation  cycles  under 
different  demands 

Trainees  learning  a 
complex  task 
should  be 
encouraged  to 
monitor  rate  of 
learning  progress 
rather  than  just 
learning 
performance 

Focusing  feedback  on  how 
and  what  KSAs  trainees 
have  developed  that 
involve  managing 
different  quantities  of 
tasks  minimizes  goal 
abandonment  and 
promotes  learning  how  to 
deal  with  situations  where 
resources  (time,  persons, 
etc.)  are  strained 

Trainees  learning  a 
complex  task 
should  be 
encouraged  to 
monitor  rate  of 
learning  progress 
rather  than  just 
learning 
performance 

Focusing  feedback  on  how  and 
what  KSAs  trainees  have 
developed  that  involve 
managing  tasks  with  fewer 
vs.  more  interdependencies 
and  considerations  minimizes 
goal  abandonment  and 
promotes  learning  how  to  deal 
with  situations  where 
resources  must  be  highly 
coordinated 

Trainees  learning  a 
complex  task 
should  be 
encouraged  to 
monitor  rate  of 
learning  progress 
rather  than  just 
learning 
performance 

Focusing  feedback  on  how 
and  what  KSAs  trainees  have 
developed  that  are  involve 
managing  sudden  changes 
in  task  demands  minimizes 
goal  abandonment  and 
promotes  learning  how  to  deal 
with  situations  where 
resources  must  be  quickly 
assessed,  gathered,  and 
distributed 

Provide  & 

emphasize  proximal 
subgoals  that  allows 
trainees  to  break 
task  down  into 

Focusing  on  how  to  deal 
with  multiple  competing 
demands  and  strained 
resources  improves 
capacity  to  manage  tasks 

Provide  & 

emphasize  proximal 
subgoals  that  allows 
trainees  to  break 
task  down  into 

Focusing  on  how  to  prioritize 
and  structure  task  activity 
improves  capacity  to  make 
informed  decisions  & 
communicate  what  must  be 

Provide  & 

emphasize  proximal 
subgoals  that  allows 
trainees  to  break 
task  down  into 

Focusing  on  how  to  deal  with 
variability  in  task 
demands/resources  within  a 
single  performance  event 
improves  capacity  to  shape 

Adapting  to  changes  in  Component 
complexity 

Changes  in  number  and/or  difficulty  of  tasks 

Adapting  to  changes  in  Coordinative  complexity 

Changes  in  sequencing,  prioritization,  & 
interdependence  among  tasks 

Adapting  to  changes  in  Dynamic  complexity 

Volatility  in  component  &  coordinative  complexity 
within  a  task 

Principle 

Rationale 

Principle 

Rationale 

Principle 

Rationale 

manageable 

components 

where  demands  >=  supply 

manageable 

components 

accomplished  to  reach  task 
goals 

manageable 

components 

and  implement  contingencies* 

Variability  in 
practice  trials  / 
simulated  clinical 
events  should  be 
provided  during 
training  to  maximize 
retention  &  transfer 

Practicing  multiple 
situations  with 
fewer/simple, 

fewer/difficult,  more/simple, 
more/difficult  exposes 
trainees  to  more 
exemplars,  prepares  them 
for  more  situations,  and 
encourages  flexible  modes 
of  thinking/problem-solving 
(Crawl-Walk-Run) 

Variability  in 
practice  trials  / 
simulated  clinical 
events  should  be 
provided  during 
training  to  maximize 
retention  &  transfer 

Practicing  multiple  situations 
with  fewer/simple, 
fewer/difficult,  more/simple, 
more/difficult  exposes  trainees 
to  more  exemplars,  prepares 
them  for  more  situations,  and 
encourages  flexible  modes  of 
thinking/problem-solving 
(Crawl-Walk-Run) 

Variability  in 
practice  trials  / 
simulated  clinical 
events  should  be 
provided  during 
training  to  maximize 
retention  &  transfer 

Practicing  situations  that 
transition  from  fewer/simple, 
fewer/difficult,  more/simple, 
more/difficult  within  the 
learning  environment  exposes 
trainees  to  more  exemplars, 
prepares  them  for  more 
situations,  and  encourages 
flexible  modes  of 
thinking/problem-solving 
(Crawl-Walk-Run) 

Trainees  should  be 
encouraged  to 
experience  errors 

Errors  of  omission  & 
commission  are  common 
stimulus  for  adaptation.* 
Placing  trainees  in 
situations  where  few  vs. 
many,  little  vs.  big,  salient 
vs.  subtle,  etc.  errors  are 
likely  and/or  have 
happened  reinforces 
situation  awareness  and 
decision-making  skills  in 
unexpected  and  unplanned 
situations 

Trainees  should  be 
encouraged  to 
experience  errors 

Errors  of  omission  commission 
are  common  stimuli  for 
adaptation.  Placing  trainees  in 
situations  where  errors  push 
them  down  a  wrong  path 
reinforces  situation  awareness 
and  decision-making  skills  in 
unexpected  and  unplanned 
situations 

Trainees  should  be 
encouraged  to 
experience  errors 

Errors  of  omission  & 
commission  are  common 
stimuli  for  adaptation.  Placing 
trainees  in  situations  where 
tasks  change  suddenly  and 
errors  are  more  likely 
reinforces  situation  awareness 
and  decision-making  skills  in 
unexpected  and  unplanned 
situations 

*TeamSTEPPS  includes  several  concepts  that  are  consistent  with  the  material  above.  Specifically,  TeamSTEPPS  supports  the  idea  of  a  “brief  where  planning 
behaviors  support  the  ability  of  teams  to  prioritize  their  work  and  develop  contingency  plans  that  facilitate  the  ability  to  adapt  quickly  in  response  to  changes. 
TeamSTEPPS  also  emphasizes  monitoring  behaviors,  which  enable  teams  to  detect  changes  that  require  them  to  adapt  their  approach.  TeamSTEPPS  also 
describes  the  need  to  monitor  team  members  to  help  prevent  errors.  Key  TeamSTEPPS  concepts  are  summarized  here: 

Brief:  Encourages  team  members  to  share  their  plan,  assign  roles  and  responsibilities,  anticipate  outcomes  and  likely  contingencies.  (Pocket  Guide,  p.  16) 
Monitoring:  TeamSTEPPS'  situation  monitoring  refers  to  monitoring  “progress  toward  goals  and  identifying  changes  that  could  alter  the  plan.”  TeamSTEPPS 
encourages  team  members  to  monitor  their  environments  for  errors.  Specifically,  situation  monitoring  includes  monitoring  “fellow  team  members  to  ensure  safety 
and  prevent  errors”  (Pocket  Guide,  p.  32) 

Leadership:  TeamSTEPPS  believes  that  effective  team  leaders  should  organize  the  team,  identify  clear  goals,  assign  tasks  and  responsibility,  monitor  and  modify 
the  plan,  communicate  changes  to  the  plan,  provide  feedback  when  needed,  manage  and  allocate  resources,  and  facilitate  information  sharing.  (Pocket  Guide,  p. 
15) 


Attachment  3.  Principles  of  providing  adaptive  feedback 


Principle  1.  Trainees  should  be  provided  with  accurate  and  credible  feedback. 

Ensuring  feedback  is  accurate  helps  trainees  understand  what  task  behaviors  need  improvement.  Making 
feedback  credible/authentic  improves  the  likelihood  that  trainees  perceive  the  feedback  as  something 
important  to  which  they  should  attend.  There  are  instances  in  which  the  accuracy  of  feedback  should  be 
"altered"  if  it  benefits  self-efficacy  and  effort  of  trainees  (e.g.,  learning  a  complex  task  that  results  in  many 
mistakes,  poor  training  performance,  etc.)  TeamSTEPPS  and  other  training  programs  support  the  provision  of 
feedback  but  do  not  provide  concrete  recommendations  to  ensure  delivery  of  adaptive  feedback. 

Simulation  Recommendations: 

•  Explain  learning  objectives  to  trainees  and  explain  clear  benchmarks  for  performance.  By  setting 
benchmarks,  trainees  can  see  where  their  performance  gaps  lie.  Setting  benchmarks  also  helps  ensure 
feedback  is  diagnostic. 

•The  feedback  facilitator  should  have  significant  skill  in  debriefing  techniques. 

•  Consider  pairing  a  content  expert  with  feedback  expert  when  needed 

Principle  2.  The  frequency  and  timing  of  feedback  should  be  appropriately  tailored  to  trainees  and  the 
goal  of  training. 

In  general,  directive,  immediate,  and  frequent  feedback  tends  to  facilitate  the  acquisition  of  declarative  & 
procedural  knowledge  and  improve  learner's  self-efficacy.  However,  when  the  goal  of  training  is  to  promote 
how  to  identify  and  handle  errors  and/or  develop  strategies  and  contingency-based  thinking,  feedback  should 
be  less  frequent  to  discourage  trainees  from  assuming  there  is  "one  correct  answer"  they  should  be  learning. 

Simulation  Recommendations: 

•  Process  feedback  should  be  more  frequent  than  outcome  feedback 

•With  more  experienced  teams,  moving  from  a  formalized  feedback  to  facilitation  of  a  high-level  debrief  that 
allows  objectives  to  emerge  based  on  performance  and  team  challenges  might  be  more  appropriate 

•  When  performing  a  more  high-level  debrief,  it  should  occur  as  close  to  the  event  as  possible 

•  Be  sure  to  build  in  adequate  time  for  debriefs,  usually  a  minimum  of  2x  the  length  of  the  simulation 

•  Ensure  that  the  simulation  objectives  are  finite  and  can  be  covered  during  the  debrief 

•  Build  in  feedback  delivery  mechanisms  into  the  Crawl-Walk-Run  training  framework 

Principle  3.  Feedback  related  to  practice  behaviors  and  clinical  performance  strategy  development 
should  be  specific. 

When  it  is  appropriate  to  provide  such  feedback  (see  principle  above),  feedback  about  the  behaviors  in  which 
trainees  engaged;  how,  why,  and  what  clinical  performance  strategies  trainees  attempted  to  implement;  and 
the  manner  by  which  they  addressed  errors  or  unexpected  events  should  be  specific  and  detailed.  Providing 
specific  feedback  facilitates  the  retention  and  automatizing  of  learned  material  and  helps  to  avoid  ineffective 
strategy  or  behavioral  changes. 

Simulation  Recommendations: 

•  Ensure  that  team  members  have  a  working  knowledge  of  team  processes  prior  to  executing  the  simulation; 
this  will  allow  the  facilitator  to  use  this  common  language  during  the  debrief 

•  Refer  to  specific  examples  during  the  simulation  to  highlight  strengths  and  weaknesses  of  team  process. 
•Video  review  may  be  helpful 

•  Providing  individuals  with  feedback  is  important;  however,  must  be  done  with  care  in  a  team  debrief 

•  Using  self-assessment  "cognitive  aids"  can  help  individuals  assess  their  contribution  to  team  performance. 
One  example  would  be  the  TeamSTEPPS  debrief  checklist  available  in  the  TeamSTEPPS  Pocket  Guide 

Was  communication  clear? 

Were  roles  and  responsibilities  understood? 

Was  situation  awareness  maintained? 

Was  workload  distribution  equitable? 

Was  task  assistance  requested  or  offered? 

Were  errors  made  or  avoided? 

Were  resources  available? 

What  went  well? 

What  should  improve? 


Principle  4.  Feedback  should  be  more  heavily  focused  towards  process  rather  than  outcome. 

Outcome  feedback  conveys  the  extent  to  which  trainees  met/are  meeting  learning  objectives.  Alternatively, 
process  feedback  focuses  on  how  trainees  are  using  information,  performing  behaviors,  and  the  steps  used  to 
complete  task  activities.  Process  feedback  directs  learners  to  reflect  on  the  strategies  and  decisions  that  led  to 
particular  outcomes,  and  is  thus  particularly  important  when  the  goal  of  training  is  to  improve 
regulatory/strategic  thinking. 

Simulation  Recommendations: 

•Allow  teams  to  discuss  medical  content  and  address  any  concerns  quickly  to  help  learners  focus  on 
processes  of  care 

•  Encourage  learners  to  consider  other  circumstances  where  similar  processes  are  employed  and  can  fail. 

This  helps  team  focus  on  processes  instead  of  the  specific  clinical  issues  presented  in  the  simulation. 

Principle  5.  Trainees  should  be  encouraged  to  believe  substantial  negative  performance  discrepancies 
are  moderate. 

Acquiring  KSAs  in  complex  task  environments  is  challenging,  and  learners  are  not  likely  to  perform  well  during 
initial  stages  of  training.  Providing  accurate  and  credible  feedback  is  important,  but  it  is  equally  critical  to 
ensure  that  trainees  do  not  become  overwhelmed  and/or  discouraged  by  actions  they  have  performed 
incorrectly.  This  balance  can  be  achieved  by  framing  feedback  such  that:  (1)  feedback  emphasizes  trainee 
performance  is  attributable  to  controllable  factors;  (2)  feedback  de-emphasizes  outcome-focused  feedback  in 
favor  of  process  feedback  and  feedback  that  highlights  how  learners  are  developing;  (3)  initially  poor 
performance  be  labeled  as  only  moderately  negative.  Doing  so  decreases  the  likelihood  of  goal  abandonment 
while  increasing  the  likelihood  that  effort  and  self-efficacy  will  be  maintained. 

Simulation  Recommendations: 

•  Encourage  learners  to  note  positive  as  well  as  negative  behaviors  (What  should  you  change?  What  should 
you  do  the  same?) 

•  Encourage  learners  to  see  how  even  effective  processes  can  result  in  poor  outcomes 

•  Limit  the  focus  of  the  debrief  to  just  learning  objectives  to  avoid  talking  about  too  many  issues 

•  Focus  on  process,  not  outcomes 

Principle  6.  The  provision  of  negative  and/or  normative  feedback  should  be  minimized  to  trainees 
learning  a  complex  task. 

Negative  feedback  (i.e.,  learners  are  failing  to  meet  learning  objectives)  and  normative  feedback  (i.e., 
comparing  learners  to  an  external  standard)  tends  to  shift  trainees'  attributions  towards  the  self  &  ego 
protection,  which  generally  interferes  with  the  acquisition  of  KSAs.  Negative  feedback-especially  when 
learning  a  complex  task--is  demotivating  and  tends  to  decrease  self-efficacy.  In  general,  positive  performance 
feedback  tends  to  improve  self-efficacy,  though  it  must  be  accurate  and  credible  to  prevent  complacency 
and/or  disengagement.  Similar  recommendations  are  noted  in  TeamSTEPPS  training  documents,  where  it 
states  feedback  should  be  timely,  respectful  (focusing  on  behaviors,  not  personal  attributes),  specific  (directed 
toward  future  improvement),  and  considerate. 

Simulation  Recommendations: 

•  Provide  a  supportive  climate  that  allows  participants  to  share  opinions  openly  and  honestly 

•  Critical  step,  as  learners  cite  a  fear  of  educator  and  peer  judgment  as  barrier 

•  Use  "good  judgment"  framework  or  advocacy/inquiry  to  discuss  negative  performance  and  uncover  learner 
mental  models  and  frames  that  are  supporting  suboptimal  performance 

Principle  7.  Guidance  that  directs  trainees  to  consider  what  they  should  think  about  and  how  to  think 
about  it  should  be  provided  to  trainees  in  learner  control  environments. 

Guidance  is  a  proactive  "feed-forward"  mechanism  that  encourages  learners  to  take  an  active  role  in 
considering  how  and  why  they  are  engaging  in  particular  learning  behaviors.  Guidance  promotes  learning 
through  both  increased  metacognition  (i.e.,  "thinking  about  thinking")  and  encouraging  an  exploratory/future- 
focused  perspective  on  learning-both  of  which  are  critical  conditions  for  learning  complex  tasks  and  strategies. 
There  are  many  options  for  what  type  of  guidance  can  be  provided,  but  typical  categories  include  focusing 
trainees  on  how  and  where  to  direct  attention  during  training  (cognition),  manage  effort  and  emotions  (affect), 
and  sequence  actions  (behaviors). 


Simulation  Recommendation: 

•  Learners  should  be  encouraged  to  identify  their  strengths  and  weaknesses.  With  instructor  input,  this 
information  should  be  used  to  guide  training  content  and  emphasis.  In  this  way,  learners  can  focus  on  more 
basic  skills  where  they  need  development  and  challenge  themselves  in  areas  where  they  excel. 

•  Guidance  can  also  come  in  the  form  of  affect/error  regulation  that  emphasizes  to  learners  that  good 
processes  don't  always  result  in  good  outcomes. 

Principle  8.  Match  the  level  of  feedback  provided  to  the  level  of  the  goals  in  training. 

Feedback  provided  in  training  directs  individuals  to  allocate  resources  and  perform  self-regulation  activities  in 
relation  to  specific  goals.  However,  trainees  can  have  goals  across  multiple  levels  thereby  complicating 
trainees'  decisions  about  which  goals  to  strive  toward.  Therefore,  if  the  focus  of  training  is  to  achieve 
individual-level  goals,  feedback  providers  should  provide  individual-level  feedback  so  resources  are  directed  to 
individual  goal  attainment.  Similarly,  if  trainees  should  focus  on  team-level  goals,  feedback  providers  should 
provide  team-level  feedback  to  direct  resources  toward  team  goal  attainment. 

Simulation  Recommendations: 

•The  debriefing  plan  should  be  pre-planned  and  should  target  appropriate  level(s)  based  upon  learning 
objectives. 

•When  individual  feedback  is  necessary  within  a  team  context,  the  learner  should  be  approached  separately  if 
there  is  an  issue  with  individual  clinical  competence  or  procedural  skills. 

•  If  individual  feedback  on  a  team  skill  is  necessary,  feedback  should  be  framed  as  a  team-based  learning 
point. 


Circulation  I  Intubation 


Attachment  4.  Variables  identified  for  use  in  BBN  Predictive  Model 


Behavioral  Type 

Team  Clinical  Behavior  or  Process 

Information  Gathering 

Assessed  pupil  reactivity 

Checks  presence  of  gag  reflex 

Attempts  to  elicit  speech 

Elicit  speech  physical 

Communication 

Communicates  information  about  signs  of  head  trauma 

Calculates  patient's  Glasgow  coma  scale 

Communicates  patient's  Glasgow  coma  scale 

Makes  decision  to  intubate  patient 

Obtains  fingerstick  glucose 

Action 

Discusses  which  intubation  medications  to  use 

Discusses  dosage  of  medications 

Gives  1  sedation  medication 

Appropriately  pretreatments  patient 

If  paralytic  used,  choice  and  dose  correct 

Orders  proper  sequence  of  drugs  for  rapid  sequence  intubation 

Stabilizes  neck  by  holding  cervical  spine  immobilization 

Preoxygenates  patient 

Team  members  follow  rapid  sequence  intubation  order 

“Bags”  patient  following  intubated 

Total  duration  of  intubation 

Monitoring 

Monitors  and  communicates  blood  pressure  during  intubation 

Monitors  and  communicates  heart  rate  during  intubation 

Monitors  and  communicates  pulse  oxygen  during  intubation 

Information  Gathering 

Verifies  endotracheal  tube  placement 

Auscultates  chest 

Checks  C02  monitor 

Evaluates  oxygen  saturation  after  intubation 

Checks  blood  pressure  after  intubation 

Orders  post  intubation  X  ray 

Interprets  post  intubation  X  ray 

Calls  radiologist  for  X  ray  clarification 

Communicates  information  about  incorrect  ETT  placement 

Decision 

Makes  decision  to  adjust  ETT  based  on  X  ray  results 

Action 

Correctly  repositions  ETT 

Orders  repeat  CXR 

Information  Gathering 

Requests  initial  vital  signs 

Confirms  IV  line  is  in  place 

Orders  cardiac  monitoring 

Undresses  patient 

Request  new/updated  vitals 

Assesses  chest  wall 

Assesses  abdominal  area 

Checks  pulse  on  arm/neck 

Assesses  back 

Communication 

Communicates  prehospital  vital  signs 

Communicates  updated  vital  signs 

Communicates  reason  for  admission 

Communicates  cardiac  rhythm 

Orders  IV  fluids 

Orders  second  IV 

Information  Gathering 

Verifies  IV  fluids  administration 

Monitors  and  communicates  blood  pressure 

Monitors  and  communicates  heart  rate 

Rhythm  assessed  to  be  “tachycardic” 

Communication 

Uses  word  "shock" 

Discusses  causes  of  hypertension 

Behavioral  Type 

Team  Clinical  Behavior  or  Process 

Action 

Orders  coagulation  studies 

Orders  type  and  cross  match 

Orders  blood  transfusion 

Orders  uncross-matched  pprbc 

Transfuses  a  minimum  of  2  units  of  uncross-matched  pprbc 

Obtains  a  surgical  consult 

Information  Gathering 

Assesses  if  blood  is  ready  for  transfusion 

Monitor  vitals  during  transfusion 

Femur 

Information  Gathering 

Checks  pulse  feet 

Communication 

Communicates  absent  right  dorsalis  pedis  pulse 

Communicates  presence  of  femur  abrasion 

Action 

Orders  femur  X-ray 

Orders  pelvis  X-ray 

Orders  head  CT 

Orders  CT  of  cervical  spine 

Obtains  FAST  exam 

Communication 

Communicates  finding  of  displaced  femur  fracture 

Communicates  finding  of  widened  symphysis  pubis  on  x-ray 

Action 

Applies  traction  to  right  leg 

Time  to  placement  of  traction 

Maintains  traction 

Checks  right  dorsalis  pulse  after  traction 

Consults  orthopedic  surgeon 

Places  pelvis  binding 

pprbc  =  prepacked  red  blood  cells 

FAST  =  focused  assessment  with  sonography  for  trauma 

CT  =  computed  tomography 

ETT  =  endotracheal  tube 
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CAPT.  Lindsay  Grubish,  DO 


POSITION  TITLE 
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Education/training 


INSTITUTION  AND  LOCATION 

DEGREE 
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year(s) 

FIELD  OF  STUDY 

Lehigh  University,  Bethlehem,  PA 

BS 

2007 

Bioengineering 

Philadelphia  College  of  Osteopathic 
Medicine,  Philadelphia,  PA 

DO 

2012 

Medicine 

National  Capital  Consortium,  Bethesda,  MD 

N/A 

2013 

Transitional  Year 
Internship 

Resident,  Madigan  Army  Medical  Center, 
Tacoma,  WA 

N/A 

2016 

Emergency  medicine 

Chief  Resident,  Madigan  Army  Medical 
Center,  Tacoma,  WA 

2015-2016 

Emergency  medicine 

Research  and  professional  experience: 

A.  Positions  and  Honors 
Positions 


2016  -  present 


Staff  physician,  Department  of  Emergency  Medicine,  Madigan  Army  Medical 
Center 


Other  Experience  and  Professional  Memberships 


2013  -  present 

2013  -  present 
2016 

2016  -  present 

2014  -  present 

Honors 

2007 

2007 

2007 

2007 

2016 

2016 


Member,  American  College  of  Emergency  Physicians 
Member,  Society  for  Academic  Emergency  Physicians 
National  Resident  Representative,  Government  Service  Chapter,  American 
College  of  Emergency  Physicians 
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Physicians 

Institutional  Review  Board,  Madigan  Army  Medical  Center,  Tacoma,  WA 


American  Society  of  Military  Engineers  Award,  Lehigh  University 

Col.  Edward  W.  Rosenbaum  Award,  Lehigh  University 

John  S.  Steckbeck  Memorial  Award,  Lehigh  University 

Joseph  P.  Hendrzak  Memorial  Award,  Lehigh  University 
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Army  Medical  Center 


Research  and  professional  experience  (continued). 

B.  Publications 

1.  Tomich,  Allison,  Grubish,  L.  "Immunocompetent,  Immunized  Male  With  Mumps,  Complicated 
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1.  Kessler,  J.,  McGrane,  K.,  Bothwell,  J.,  Grubish,  L.,  Implementation  of  Tactical  Breathing 
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Simulated  Cadaveric  Model  Comparison  of  Intubation  using  Personal  Protective  Equipment  and 
Standard  Uniform  in  a  Simulated  Cadaveric  Model,  Military  Health  System  Research 
Symposium,  2016,  Orlando,  FL. 

3.  Grubish,  L.  The  use  of  model-based  iterative  reconstruction  to  decrease  ED  radiation  exposure. 
Scientific  Assembly  of  American  College  of  Emergency  Physicians,  2014,  Chicago,  IL. 

4.  Grubish,  L.  Comparison  of  Intubation  Performance  by  Emergency  Medicine  Residents  Using 
Video  Laryngoscopy  versus  Direct  Laryngoscopy  in  a  Simulated  Angioedema  Cadaveric  Model. 
Scientific  Assembly  of  American  College  of  Emergency  Physicians,  2015,  Boston,  MA. 
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Education 

6/2001-5/2005 

8/2005-5/2009 

8/2012-present 


Bachelor  of  Science,  cum  laude,  Biology,  University  of  Utah,  Salt  Lake 
City,  UT 

Doctor  of  Medicine,  Saint  Louis  University  School  of  Medicine,  St.  Louis, 
MO 

Masters  of  Science,  Clinical  Research,  Indiana  University  School  of 
Medicine,  Indianapolis,  IN 


Postgraduate  Training 

7/2009-6/2012  Emergency  Medicine  Residency,  Indiana  University  School  of  Medicine, 

Indianapolis,  IN 


6/2011-7/2012  Chief  Resident,  Indiana  University,  Indianapolis,  IN 

7/2012  Systems  Engineering  Initiative  for  Patient  Safety,  Short  Course, 

University  of  Wisconsin  -  Madison,  Madison,  WI 


Faculty  Positions 

7/2012-7/2014  Assistant  Professor  of  Clinical  Emergency  Medicine,  Indiana  University 

School  of  Medicine,  Indianapolis,  IN 

7/2014-present  Acting  Instructor,  Division  of  Emergency  Medicine,  University  of 
Washington  School  of  Medicine,  Seattle,  WA 


Hospital  Positions 

7/20 12-7/20 1 4  Attending  Physician,  Emergency  Medicine 

Indiana  University  Health 
Methodist  Hospital 
University  Hospital 

7/2014-present  Attending  Physician,  Emergency  Medicine,  University  of  Washington 
Medical  Center,  Seattle,  WA 


7/2014-present  Attending  Physician,  Emergency  Medicine,  Harborview  Medical  Center, 

Seattle,  WA 


Honors 


5/200 1  National  Merit  Finalist,  University  of  Utah 

8/2001-5/2005  Presidential  Scholar,  University  of  Utah 

7/2006-5/2009  Gardner  Scholarship,  Saint  Louis  University  School  of  Medicine 

7/2007-5/2009  Jacobs  Scholarship,  Saint  Louis  University  School  of  Medicine 

7/2008  Arnold  Gold  Humanism  in  Medicine  Award,  Saint  Louis  University 

School  of  Medicine 

2/2012  Red  Shoes  Award,  Riley  Hospital  for  Children,  Indiana  University, 

Department  of  Pediatrics 

3/2012  Best  Poster  Presentation  in  Senior  Academic  Faculty  Category,  Council  of 

Emergency  Medicine  Residency  Directors  Annual  Meeting 

6/2012  Resident  Teacher  of  the  Year,  Indiana  University,  Department  of 

Emergency  Medicine 

10/2012  Honorable  Mention  Award,  Quality  Improvement  and  Patient  Safety 

Section,  American  College  of  Emergency  Physicians  Annual  Meeting 

6/20 1 3  Faculty  Impact  Award,  Indiana  University,  Department  of  Emergency 

Medicine 

2015  PRAISE  Award,  Harborview  Medical  Center 

6/2016  Outstanding  Educator  Award  Nominee,  University  of  Washington 

Division  of  Emergency  Medicine 

Board  Certification 

6/2013,  2023  Diplomate,  American  Board  of  Emergency  Medicine  (ABEM) 

Current  Licenses  to  Practice 

10/2011  -6/20 1 4  Indiana  State  Medical  License 

3/2014-present  Washington  State  Medical  License 

Professional  Organizations 

2009-20 1 5  Member,  American  Academy  of  Emergency  Medicine 

2009-2015  Member,  American  College  of  Emergency  Physicians 

2009-present  Member,  Society  of  Academic  Emergency  Medicine 

Teaching  Responsibilities 

1/2010  Lecturer,  Mini-Lessons  in  Patient  Safety:  Communication  Failures, 

Department  of  Emergency  Medicine,  Indiana  University 

5/2010  Lecturer,  Mini-Lessons  in  Patient  Safety:  Cognitive  Forcing  Strategies, 

Department  of  Emergency  Medicine,  Indiana  University 


6,  10/2010,  5/2011 

Instructor,  MS4  Suture  Lab,  Department  of  Emergency  Medicine,  Indiana 
University 

1/2011,4/2011 

Instructor,  Simulation  Training,  Department  of  Emergency  Medicine, 
Indiana  University 

4/2011 

Lecturer,  Mini-Lessons  in  Patient  Safety:  Introduction  to  the  Near  Miss 
Blitz,  Department  of  Emergency  Medicine,  Indiana  University 

5/2011 

Lecturer,  Approach  to  Head  Injuries,  Indianapolis  Emergency  Medical 
Services 

6/2011 

Lecturer,  Approach  to  Sepsis,  Indianapolis  Emergency  Medical  Services 

6/2011 

Lecturer,  Acute  Upper  Airway  Disorders  and  Obstruction:  EM  Grand 
Rounds,  Department  of  Emergency  Medicine,  Indiana  University 

8/2011 

Lecturer,  On  Beyond  Zebra:  EM/Peds  Combined  Grand  Rounds, 
Combined  Departments  of  Emergency  Medicine  and  Pediatrics,  Indiana 
University 

8/2011 

Lecturer,  The  Violent  Patient,  Department  of  Emergency  Medicine, 
Indiana  University 

3/2012 

Lecturer,  Updates  to  ACLS  2012:  EM  Grand  Rounds,  Department  of 
Emergency  Medicine,  Indiana  University 

3/2012 

Lecturer,  Hypothermia,  Indianapolis  Emergency  Medical  Services 

5/2012 

Presenter,  The  Near  Miss  Blitz,  Indiana  American  College  of  Emergency 
Physicians  Annual  Meeting 

6/2012 

Presenter,  Patient  Safety  and  Graduate  Medical  Education,  Graduate 
Medical  Education  Committee  Meeting,  Indiana  University 

7/2012 

Panelist,  Expert  panel,  RN-Physician  Communication,  Department  of 
Emergency  Medicine,  Indiana  University 

9/2012,  5/2013 

Instructor,  Pediatric  Emergency  Simulation  session,  Combined 
Departments  of  Emergency  Medicine  and  Pediatrics,  Indiana  University 

1,5,  11/2012 

Instructor,  Simulation  Training,  Department  of  Emergency  Medicine, 
Indiana  University 

2/2013,2/2014 

Course  Facilitator,  Evidence  Based  Medicine,  Indiana  University  School 
of  Medicine 

3/2013 

Lecturer,  Nurse-Physician  Communication:  EM  Grand  Rounds, 
Department  of  Emergency  Medicine,  Indiana  University 

4/2013,  8/2013 

Lecturer,  Approach  to  the  Febrile  Child,  Department  of  Emergency 
Medicine,  Indiana  University 

7/2013 

9/2013 

6/2013,  11/2013 
1/2014 

2/2013,2/2014 

10/23/2014 

5/2015 

10/2015 

10/2014,  5/2015 

11/2015 

10/2015 

7/2015-6/2018 

1/7/2016 


Lecturer,  Cognitive  Error  and  ED  Providers:  The  Emergency  Department 
as  the  Perfect  Storm  for  Medical  Error,  Intern  Orientation,  Department  of 
Emergency  Medicine,  Indiana  University 

Lecturer,  Research  Bootcamp  for  Junior  Faculty:  Building  a  Database, 
Department  of  Emergency  Medicine,  Indiana  University 

Instructor,  Simulation  Training,  Department  of  Emergency  Medicine, 
Indiana  University 

Instructor,  Simulation  Training,  Department  of  Emergency  Medicine, 
Indiana  University 

Course  Facilitator,  Evidence  Based  Medicine,  Indiana  University  School 
of  Medicine 

Lecturer,  A  Motley  Crew:  Collagen  Vascular  Diseases  and  Acute 
Immunologic  Disorders,  Emergency  Medicine  Grand  Rounds,  University 
of  Washington 

Instructor,  ISIS  Capstone:  Team  Based  Interprofessional  Training 
Simulation,  University  of  Washington  School  of  Medicine 

Instructor,  Mock  Oral  Boards,  Emergency  Medicine  Residency  Program, 
University  of  Washington 

Instructor,  Medical  Student  Simulations:  Dyspnea,  Chest  pain,  ACLS 

Airway,  University  of  Washington  School  of  Medicine 

Instructor,  Resuscitation  Capstone  Course,  Emergency  Medicine 
Residency  Program,  University  of  Washington 

Co-Mentor  and  Lecturer,  Continuous  Quality  Initiative  project,  Class  of 
2018,  Emergency  Medicine  Residency  Program,  University  of 
Washington 

Lecturer,  Understanding  Prognosis,  Evidence  Based  Medicine  Series, 
Emergency  Medicine  Residency  Program,  University  of  Washington 


9/2016  Co-lecturer,  Evidence  Based  Medicine  Core  Faculty  group,  Clinical 

Decision  Rules,  Emergency  Medicine  Residency  Program,  University  of 
Washington 


Editorial  Responsibilities 

None 


Special  National  Responsibilities 


3/2012-3/2013 


Chair,  Resident  and  Student  Advisory  Committee,  Society  of  Academic 
Emergency  Medicine 


Special  Local  Responsibilities 

6/2010-6/2011  Chair,  Patient  Safety  Committee,  Department  of  Emergency  Medicine, 

Indiana  University 

6/2010-6/2011  Member,  Orientation  and  Recruitment  Committee,  Department  of 

Emergency  Medicine,  Indiana  University 

6/20 1 0-6/20 1 1  Member,  Medical  Student  Committee,  Department  of  Emergency 

Medicine,  Indiana  University 

10/2010  Mentor,  pre -medical  interest  group,  Butler  University,  Indianapolis,  IN 

10/2010-10/2011  Member,  35th  Anniversary  Gala  Planning  Committee,  Department  of 
Emergency  Medicine,  Indiana  University 

1-6/2011  Chair,  Senior  Banquet  Committee,  Department  of  Emergency  Medicine, 

Indiana  University 

4/2012  Supervising  Physician,  Student  Outreach  Community  Clinic,  Indiana 

University  School  of  Medicine 

6/2012-5/2014  Mentor,  Dual  Medical  Marriages,  Department  of  Emergency  Medicine, 

Indiana  University 

6/2012-5/2014  Mentor,  Women  in  Emergency  Medicine,  Department  of  Emergency 

Medicine,  Indiana  University 

7/2012-5/2014  Chair,  Resident/Fellow  Patient  Safety  Council  Planning  Group  for  the 

Graduate  Medical  Education  Office,  Indiana  University  School  of 
Medicine 

7/2012-5/2014  Advisor,  medical  student  career  advising,  Indiana  University  School  of 

Medicine 

9/2014-present  Member,  Emergency  Department  Quality  Assurance  Committee, 
Harborview  Medical  Center 

9/2014-present  Member,  Emergency  Department  Safety  Committee,  Harborview  Medical 
Center 

4/20 15 -present  Member,  Emergency  Medicine  Operations  and  Policy  Research 
Workgroup,  University  of  Washington  Emergency  Medicine 

4/20 15 -present  Member,  Evidence  Based  Emergency  Medicine  Faculty  Core,  University 

of  Washington  Emergency  Medicine 

9/2015  Advisor,  Career  Night,  Emergency  Medicine  Residency  Program, 

University  of  Washington 

7/2016-present  Advisor,  Women  in  Emergency  Medicine,  Emergency  Medicine 
Residency  Program,  University  of  Washington 


Research  Funding 

7/2016  -  3/2018  Washington  State  Labor  and  Industries  Department,  Safety  and  Health 
Investments  Projects  (SHIP)  Award,  Preventing  Violence  against 
Emergency  Department  Healthcare  Workers:  A  Prospective  Needs 
Assessment  to  Inform  Effective  Intervention 
$200,000 

1/2017  -  1/2018  University  of  Washington,  Department  of  Internal  Medicine,  Patient 

Safety  Acclerator  Grant  Award,  Piloting  a  Mobile  Application  for  Wound 

Care  Follow  Up  in  the  Emergency  Department 

$20,000 
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Abstracts 
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Education  Annual  Educational  Conference,  February  2012*. 
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Lectures  and  Presentations 

1.  5/2013.  How  to  Become  a  Trailblazer:  Perspectives  of  Resident  Innovators.  Society  for 
Academic  Emergency  Medicine  Annual  Meeting. 

2.  2,  5/2014.  A  CLER  Plan:  Implementing  a  Patient  Safety  Council  for  Residents  and 
Fellows.  Accreditation  Council  for  Graduate  Medical  Education  Annual  Educational 
Conference,  February  2014;  Society  for  Academic  Emergency  Medicine,  Annual 
Meeting,  May  2014. 

3.  3/2015.  Resident  Led  Patient  Safety,  Council  of  Emergency  Medicine  Residency 
Directors  Academic  Assembly 

4.  1/2016.  Seeing  Red:  Navigating  the  Combative  Patient.  An  Innovative  De-escalation 
Curriculum  for  the  Provider,  International  Meeting  for  Simulation  in  Healthcare. 


Other  Service 

2-5/2010  Member,  Indianapolis  Bike  Safety  Fair  for  Kids  Planning  Committee, 

Indianapolis,  IN 

1 1/2011  Mentor,  Photovoices  Project,  Indianapolis,  IN 

•  Mentored  at-risk  children  and  assisted  them  in  expression  and 
communication  via  photography. 

2-4/2012  Mentor,  Team  Triathlon,  Indianapolis,  IN 

•  Coached  and  mentored  9  inner-city  children  in  training  for  and 
completing  a  kids  ’  triathlon. 

4/2012  Volunteer,  Indy  Connects  physician,  Indianapolis,  IN 

•  Provided  free  medical  care  for  the  homeless  during  community 
fair. 
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Development  of  an  Integrated  Team  Training  Design  and  Assessment 
Architecture  to  Support  Adaptability  in  Healthcare  Teams 

MSIS-Team  Performance  Training  Research  Initiative 

PI:  R.  Fernandez  /  J.  Grand  Org:  University  of  Washington  PY2  Annual  Report 


Problem,  Rationale,  and  Military  Relevance 

•  Problem:  Conceptual  models  and  assessment  approaches  to 
support  effective  team  training  that  maximizes  team  adaptability 
and  performance  do  not  exist. 

•  Rationale:  An  integrated  team  training  model  will  identify  which 
individual,  team,  and  training  design  factors  can  be  manipulated 
to  maximize  team  training  effectiveness  and  impact  on  patient 
safety  outcomes.  Additionally,  a  predictive  model  of  team 
performance  will  demonstrate  how  team  behaviors  predict  future 
team  performance  and  patient  care  outcomes. 

•  Military  Relevance:  This  proposal  directly  addresses  the  TPT 
research  initiative  by  providing  a  detailed  framework  and 
predictive  assessment  system  to  support  team  performance 
training  to  improve  teamwork  behaviors  and  patient  outcomes. 

Proposed  Solution 

•  Objective:  To  develop  a  simulation  design  architecture  and 
predictive  model  of  trauma  team  performance  to  support  team 
training  and  team  effectiveness. 

•  Summary  of  Aims:  Integrate  individual-  and  team-level  team 
performance  frameworks  to  develop  a  simulation  design 
architecture  and  a  predictive  model  of  trauma  team 
performance  to  support  effective  team  training  with  automated 
individual  and  team  feedback  and  performance  assessment. 

•  Outcomes:  (1 )  A  detailed  framework  of  the  individual,  team, 
and  training  design  factors  related  to  effective  team 
performance  training  and  (2)  A  predictive  model  of  team 
performance  that  identifies  how  teams  can  adapt  their 
behaviors  to  maximize  their  teamwork  and  minimize  errors 


Conceptual  Model  of  Predictive  Model  of 

Team  Training  Elements  I  Team  Performance 


IMPROVED  ADAPTABILITY  AND 
PATIENT  SAFELY 


Timeline  and  Cost  (Expenditures  to  Date  =  $450 K) 


Activities  FY 

15 

16 

NCE 

X 

Integrate  individual-level  and  team-level 
simulation  design  frameworks  to  develop  a 
simulation  design  architecture  (Aim  1) 

X 

Develop  a  predictive  model  of  trauma  team 
performance  and  outcomes  using  Bayesian  Belief 
Networks  (Aim  2) 

Prospectively  test  and  refine  the  model  of  trauma 
team  performance  on  simulated  trauma  team 
resuscitations  (Aim  2) 

Data  analysis  and  dissemination 

Estimated  Budget  ($K) 

591 

556 

Actual  Expenditures  ($K) 

170 

384 

594 

